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ABSTRACT 

The  Navy  is  currently  testing  a  new  source  selection  tool 
known  as  Red  Yellow  Green  (RYG) .  RYG  helps  field  contracting 
activities  obtain  the  best  purchase  value  by  adding  the  costs 
of  poor  contractor  performance  into  the  source  selection 
equation.  The  program  should  improve  the  quality  of  material 
received  by  the  Government  and  the  performance  of  Government 
contractors.  RYG  can  be  used  in  conjunction  with  Blue  Ribbon 
Contractor  programs  and  moves  field  contracting  activities 
toward  Total  Quality  Management  implementation. 
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I.   INTRODUCTION 

A.   BACKGROUND 

The  Navy  is  currently  testing  a  new  source  selection  tool 
known  as  Red  Yellow  Green  (RYG) .  The  test  is  being  conducted 
at  five  Navy  field  activities  (Naval  Air  Engineering  Center 
Lakehurst,  Naval  Avionics  Center  Indianapolis,  Naval  Ships 
Parts  Control  Center  Mechanicsburg  (Code  021,  Level  1/SS) , 
Naval  Supply  Center  Charleston/Naval  Shipyard  Charleston,  and 
Naval  Supply  Center  Pensacola/Naval  Aviation  Depot  Pensacola) . 
The  purpose  of  the  RYG  program  is  to  assist  Government 
contracting  personnel  in  obtaining  the  best  purchase  value  by 
adding  the  costs  of  poor  contractor  performance  into  the 
source  selection  equation  [Ref .  1]  .  The  RYG  program  was 
designed  especially  for  use  by  field  contracting  activities 
[Ref.  2]. 

In  addition  to  RYG,  several  other  contractor  quality 
programs  are  currently  being  employed  or  developed  within  the 
Department  of  Defense  (DOD)  that  consider  contractor  quality 
and  past  performance  in  the  source  selection  process.  Three 
notable  quality  programs  are  the  Blue  Ribbon  Contractor 
program,  the  Contractor  Performance  Assessment  Reporting 
System  (CPARS) ,  and  Total  Quality  Management  (TQM) .  This 
thesis  will  describe  the  four  programs  listed  above.  It  will 
then  evaluate  the  RYG  program  by  comparing  and  contrasting  it 


to  the  other  three  programs,  discussing  its  strengths  and 
weaknesses,  and  describing  the  status  of  the  program  test. 

B.   OBJECTIVES 

DOD  and  its  agencies  have  continuously  sought  ways  to 
improve  the  quality  of  Government  contractor  performance  while 
complying  with  the  statutes,  regulations,  and  policies  that 
govern  the  Federal  acquisition  process.  Recent  problems  with 
the  quality  of  material  delivered  to  DOD,  many  of  them 
highlighted  in  the  press,  have  increased  DOD's  emphasis  on 
contractor  quality  management.  At  the  same  time,  steadily 
increasing  Congressional  oversight  in  the  form  of  more  and 
increasingly  detailed  statutes  and  guidance  have  made 
development  of  a  viable  program  more  difficult.  Some  of  the 
programs  currently  employed  or  under  review  by  DOD  commands 
include  the  Navy's  RYG  program.  Blue  Ribbon  Contractor 
programs  within  the  Navy  and  Air  Force,  the  Air  Force's  CPARS, 
and  the  DOD-wide  TQM  initiative. 

One  of  these  programs,  the  Navy's  RYG,  is  the  subject  of 
this  thesis.  This  program  is  intended  to  eventually  be 
extended  to  all  DOD  agencies.  It  uses  the  data  contained  in 
the  Navy's  newly  established  unified,  centralized  Contractor 
Evaluation  System  (CES) .  These  data  are  compiled  from  many 
sources  including  Quality  Deficiency  Reports  (QDRs)  and  the 
Naval  Material  Quality  Assessment  Office  (NMQAO) .  [Ref.  3] 
Prior  to  CES,  these  data  were  found  in  many  different  forms 


and  each  came  from  a  different  source.  Now,  they  are 
available  in  a  single  location  and  can  be  accessed  by  field 
procurement  activities  from  a  computer  terminal  using  an  on- 
line call-up  menu.  CES  is  used  to  classify  contractors  (by 
Federal  Supply  Class  (FSC) )  as  high  risk  quality  performers 
(Red)  ,  moderate  risk  (Yellow)  ,  or  low  risk  (Green)  . 
Contractors  who  do  not  meet  established  criteria  for  RYG 
classification  are  listed  in  the  "Insufficient  Data" 
category.  [Ref.  l:p.  1]  The  RYG  classification  is  used  by 
contracting  personnel  to  either  apply  Technical  Evaluation 
Adjustments  (TEAs)  to  contractor  proposals  [Ref.  l:p.  1]  or, 
if  the  Fixed  Price-Greatest  Value  method  is  employed,  to  rate 
the  offerors  in  terms  of  expected  quality  of  performance  [Ref. 
l:encl  (3),  p.  3].  The  TEAs  represent  the  anticipated  cost  to 
the  Government  to  correct  or  take  appropriate  action  due  to 
poor  contractor  performance  [Ref.  l:encl  (1),  pp.  1-2]. 
Proposals  including  TEAs  are  used  to  help  determine  a  source 
selection  that  will  result  in  the  best  overall  price  to  the 
Government  [Ref.  Irencl  (1),  p.  6], 

The  RYG  program  is  undergoing  a  one-year  test  coordinated 
by  the  Office  of  Assistant  Secretary  of  the  Navy  (Shipbuilding 
and  Logistics)  Reliability,  Maintainability,  and  Quality 
Assurance  (ASN (S&L) RM&QA)  which  began  1  August  1989  using  a 
limited  number  of  commodities. 


C.  THE  RESEARCH  QUESTIONS 

The  primary  research  question  is:  How  is  the  Navy's  Red 
Yellow  Green  program  structured  and  how  is  it  intended  to 
improve  the  quality  of  material  procured  by  the  Navy?  The 
subsidiary  research  questions  are: 

1.  How  are  Red,  Yellow,  and  Green  contractors  defined? 

2.  What   are   the   other   current   contractor   quality 
initiatives  within  DOD? 

3 .  How  does  RYG  compare  and  contrast  to  these  other 
programs? 

4.  What  are  the  strengths  and  weaknesses  associated  with 
RYG? 

5.  How  is  the  RYG  program  being  tested  in  the  market? 

D.  SCOPE  AND  LIMITATIONS 

The  purpose  of  this  thesis  is  to  evaluate  the  potential 
effectiveness  of  the  Navy's  RYG  program.  It  will  analyze  the 
program  in  detail  and  describe  other  DOD  contractor  quality 
initiatives  in  brief.  It  will  compare  and  contrast  the  RYG 
program  to  the  other  DOD  initiatives  and  will  attempt  to 
identify  RYG  strengths  and  weaknesses.  It  will  evaluate  the 
success  of  RYG  tests  as  far  as  they  have  progressed.  Finally, 
it  will  consider  the  viability  of  the  RYG  program  in  the 
future,  both  within  the  Navy  and  DOD. 

The  research  effort  was  limited  by  the  short  period  of 
time  over  which  the  test  of  RYG  could  be  evaluated.  Although 
some  test  activities  were  able  to  begin  implementation  of  the 
RYG  program  on  1  August  1989,  others  were  delayed.    This 


thesis  evaluated  the  program  very  early  in  the  test  period, 
and,  therefore,  relied  on  limited  data  currently  available. 

E.  METHODOLOGY 

Two  methods  were  used  to  collect  data  and  answer  the 
research  questions.  Data  were  collected  from  secondary 
sources  to  obtain  information  about  RYG  and  other  DOD  contrac- 
tor quality  initiatives.  Data  were  collected  from  primary 
sources,  specifically  through  telephone  and  personal  inter- 
views, to  add  to  the  above  areas  and  to  assess  the  current 
status  of  the  test  and  the  potential  for  future  expansion  of 
the  program.  The  interviews  involved  personnel  at 
ASN(S&L)RM&QA,  NMQAO,  the  Naval  Supply  Systems  Command,  and 
the  five  activities  involved  in  the  test. 

F.  ORGANIZATION  OF  THE  STUDY 

The  remainder  of  the  thesis  is  organized  into  the  follow- 
ing chapters: 

1.  Chapter  II,  "Red  Yellow  Green  Program,"  will  describe 
the  RYG  program  concept  and  how  it  is  being  coordinated. 

2.  Chapter  III,  "Other  Department  of  Defense  Quality 
Initiatives,"  will  briefly  describe  other  current  DOD 
contractor  quality  programs  in  place  or  under  review. 

3.  Chapter  IV,  "A  comparison  of  Red  Yellow  Green  and  Other 
Contractor  Quality  Initiatives,"  will  compare  and 
contrast  the  RYG  program  to  the  quality  programs 
described  in  Chapter  III. 

4.  Chapter  V,  "Strengths  and  Weaknesses  of  Red  Yellow 
Green,"  will  describe  the  advantages  and  potential 
problems  and  limitations  associated  with  RYG.  It  will 
also  discuss  the  status  of  the  RYG  test,  describing  how 


the  program  test  is  progressing  and  attempting  to 
project  how  the  program  will  fare  in  the  future. 

Chapter  VI,  "Conclusions  and  Recommendations,"  will 
discuss  whether  RYG  will  be  effective  in  improving  the 
quality  of  material  procured  by  the  Navy  and  will 
comment  on  possible  changes  for  weak  areas  of  the 
program. 


II.   RED  YELLOW  GREEN  PROGRAM 

A .  BACKGROUND 

The  Navy's  Contractor  Evaluation  System  (CES)  and  Red 

Yellow  Green  (RYG)  program  were  established  to  both  assist 

Navy  activities  in  procuring  quality  goods  and  services  and  to 

comply  with  Department  of  Defense  (DOD)  and  Secretary  of  the 

Navy  (SECNAV)  policy.    Specifically,  DOD  Directive  4155.1 

states: 

DOD  Components  shall  assure  that  contracts  are  not  awarded 
to  contractors  with  a  previous  history  of  providing  products 
or  services  of  an  unsatisfactory  quality.  Contractor 
quality  history  data  shall  be  maintained  and  used  for  this 
purpose.   [Ref.  3:p.  1-1] 

Further,  SECNAV  Instruction  4855.7  states: 

It  is  Secretary  of  the  Navy  policy  that: 

a.  Contractor  quality  history  data  be  collected  and 
maintained  in  a  single  Navy  CES. 

b.  Contractor  quality  history  data  be  used  to  assure  that 
contracts  are  not  awarded  to  contractors  with  a  previous 
history  of  providing  products  of  an  unsatisfactory  quality, 
and  to  determine  actions  necessary  before  and  after  contract 
awards  to  assure  product  quality.   [Ref.  4] 

The  CES  and  RYG  program  satisfy  the  requirements  of  these 

policies . 

B.  CONTRACTOR  EVALUATION  SYSTEM 

The  CES  is  currently  composed  of  data  from  the  Product 
Deficiency  Reporting  and  Evaluation  Program  (PDREP)  data  base. 
The  Navy  has  plans  to  expand  the  CES  to  include  other  general 
information  about  contractors  such  as   financial  data   in 


addition  to  information  concerning  contractor  quality.   This 

expanded  CES  will  be  called  the  Buyer  Information  System.   It 

will  be  available  to  Government  contracting  personnel  to  help 

them  better  evaluate  offerors.    At  present,  however,  the 

CES/PDREP  is  a  quality  reporting  system  managed  by  the  Naval 

Sea   Systems   Command   (NAVSEA)   Detachment,   Naval  Material 

Quality  Assessment  Office  (NMQAO) ,  under  the  direction  of  the 

Office  of  the  Assistant  Secretary  of  the  Navy  (Shipbuilding 

and   Logistics)   Reliability,   Maintainability,   and  Quality 

Assurance  (ASN (S&L) RM&QA) .   [Ref.  3:p.  v] 

PDREP  was  established  to  satisfy  the  requirements  of 

SECNAV  Instruction  4855. lA  for  "a  product  deficiency  reporting 

and  data  feedback  system,  maintenance  of  contractor/supplier 

quality  history  and  effective  use  of  the  data  to  influence  the 

pre-contract   award  process   and   formulate  the  basis   for 

necessary  post  award  quality  assurance  actions."    [Ref.  5] 

The  purpose  of  PDREP  is  to 

. . . implement  a  product  deficiency  reporting  and  evaluation 
program  that  will  feed  data  back  to  those  activities  respon- 
sible for  design,  development,  purchasing,  production, 
supply,  maintenance,  contract  administration,  and  other 
acquisition  functions  and  initiate  action  to  appraise, 
correct,  and  prevent  product  deficiencies,  decrease  material 
ownership  costs,  evaluate  contractor/supplier  performance 
and  use  performance  data  for  making  procurement  decisions. 
[Ref.  5:p.  2] 

PDREP 's  purpose,  then,  is  compatible  with  the  goals  of  the  RYG 

program.   Therefore,  it  was  a  logical  data  base  to  be  chosen 

for  the  Navy's  new  CES  which  was  intended  to  be  used  to 


categorize  contractors  as  Red,  Yellow,  or  Green  within  a  given 
Federal  Supply  Class  (FSC) . 

The  CES  is  composed  of  contractor  quality  information 
collected  from  the  sources  which  are  described  below. 

1.  Quality  Deficiency  Reports  (QDRs) .  QDRs  are  prepared  by 
Navy  field  activities  to  document  product  quality 
deficiencies,  design  deficiencies,  or  inadequate 
procurement  documents  resulting  in  defective  new  and 
newly  reworked  material  being  delivered  to  the  Navy 
[Ref.  6].  All  QDRs  are  submitted  to  Naval  Air  Systems 
Command  (NAVAIR) ,  the  Navy  focal  point  for  QDRs.  Once 
each  week  QDRs  determined  to  be  contractor  liable  and 
with  defects  verified  are  transmitted  by  NAVAIR  to  NMQAO 
for  inclusion  in  PDREP.   [Ref.  7] 

2.  Material  Inspection  Record  (MIR).  MIRs  are  prepared 
either  by  Navy  representatives  performing  technical 
inspections  at  a  contractor's  plant  or  by  Navy  field 
activities  performing  technical  inspections  upon  receipt 
of  material.  MIRs  are  submitted  to  the  Navy  Systems 
Command  having  cognizance  over  the  field  activity. 
[Ref.  3:p.  1-5]  The  Systems  Commands  (NAVAIR,  NAVSUP, 
Naval  Facilities  Engineering  Command,  and  Space  and 
Naval  Warfare  Command)  then  transmit  the  MIRs  to  NMQAO. 
The  MIRs  generated  by  NAVSEA  activities  are  submitted 
directly  to  NMQAO.   [Ref.  7] 

3.  Reports  of  Discrepancy  (RODs) .  RODs  are  prepared  by 
Navy  field  activities  to  document  receipt  of  incorrect 
material,  shortages  and  overages,  and  discrepancies  in 
preservation,  packing,  and  marking  [Ref.  6:p.  1-2]. 
RODs  are  submitted  to  Naval  Supply  Systems  Command 
(NAVSUP) .  Procedures  are  currently  being  developed  to 
transmit  RODs  from  NAVSUP  to  NMQAO  for  inclusion  in 
PDREP.  However,  at  present  they  are  not  being 
transmitted  and,  therefore,  are  not  yet  part  of  PDREP. 
[Ref.  7] 

4.  Defense  Logistics  Agency  (DLA)  Contractor  Improvement 
Program  (CIP)  Alert  List.  Contractors  are  placed  on  the 
DLA  alert  list  if  DLA  has  placed  them  in  the  CIP,  if 
they  have  received  a  negative  pre-award  survey,  or  if 
Defense  Contract  Administration  Services  (DCAS)  has 
recommended  they  be  given  a  pre-award  survey  for  a 
particular  reason  [Ref.  8].  DLA  sends  a  hard  copy  of 
the  list  to  ASN(S&L)RM&QA.  ASN  then  sends  a  copy  to 
NMQAO.   [Ref.  7] 


5.  DLA  Method  C,  D,  and  E  Corrective  Action  Listing. 
Contractors  are  placed  on  the  corrective  action  listing 
if  DLA  has  documented  deficiencies  in  their  quality 
programs.  Specifically,  method  C  indicates  that  the 
contractor  has  a  serious  quality  problem  or  has  not 
corrected  a  deficiency  documented  using  method  B  (a 
major  deficiency) .  The  Government  sends  a  letter  to  the 
firm's  top  management  requesting  corrective  action. 
Method  D  indicates  that  less  severe  methods  of 
corrective  action  (i.e.,  A,  B,  and  C)  have  failed.  The 
acquisition  quality  assurance  program  is  discontinued, 
and  the  contractor  is  advised  that  the  Government  will 
not  accept  his  goods  or  services  until  deficiencies  have 
been  corrected.  Method  E  is  used  to  advise  a  prime 
contractor  that  a  subcontractor  has  quality  deficiencies 
that  would  justify  method  C  or  D  corrective  action  in  a 
prime  contractor  and  to  request  that  the  prime  take 
corrective  action  with  his  subcontractor.  [Ref.  9]  DLA 
sends  a  hard  copy  of  the  listing  to  ASN(S&L)  RM&QA. 
ASN  then  sends  a  copy  to  NMQAO.   [Ref.  7] 

6.  Navy  Vendor  Data  Analysis  Report  (VDAR)  .  The  VDAR 
identifies  contractors  who,  because  of  past  poor 
performance,  should  be  considered  carefully  before  being 
awarded  a  contract  and  should  be  monitored  after 
contract  award.  Evaluation  of  performance  is  based  on 
data  from  pre-award  surveys,  product  oriented  surveys, 
quality  system  reviews,  QDRs,  open  DLA  method  C,  D,  or 
E  corrective  action,  and  conviction  of  or  investigation 
for  malpractice  or  fraud.  [Ref.  8:p.  E-5]  The  VDAR  is 
compiled  by  NMQAO  based  on  past  performance  and  input 
from  Navy  Systems  Commands  and  their  field  activities 
[Ref.  10]. 

7.  Pre-Award  Surveys.  Pre-award  surveys  are  conducted  by 
contract  administration  offices  when  a  procuring 
contracting  officer  needs  additional  information  to 
determine  contractor  responsibility.  The  survey 
evaluates  the  contractor's  management,  financial 
capability,  and  technical  capability  to  determine 
whether  he  will  be  able  to  perform  the  proposed 
contract.  [Ref.  11]  Only  those  pre-award  surveys 
requested  by  Navy  activities  are  included  in  PDREP.  The 
Navy  activity  that  requested  the  survey  submits  a  copy 
to  the  cognizant  Systems  Command.  The  Systems  Commands 
then  transmit  copies  to  NMQAO.  [Ref.  10]  NAVSEA 
activities  submit  copies  of  surveys  directly  to  NMQAO 
[Ref.  7]. 

8.  Product-Oriented  Surveys.  Product-oriented  surveys  are 
technical    product    inspections    conducted    in    a 
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contractor's  plant  when  a  buying  activity  desires  to 
perforin  a  special  test  on  an  item.  They  are  performed 
by  DCAS  when  requested  by  the  buying  activity.  If  DCAS 
does  not  have  the  necessary  resources,  the  buying 
activity  may  perform  the  survey.  When  a  Navy  activity 
requests  a  product-oriented  survey,  it  submits  a  copy  to 
the  appropriate  Systems  Command.  The  Systems  Commands 
then  transmit  the  surveys  to  NMQAO.  [Ref.  10]  NAVSEA 
activities  submit  copies  of  surveys  directly  to  NMQAO 
[Ref.  7]. 

9.  Quality  System  Reviews.  Quality  system  reviews  are 
performed  by  DCAS.  They  involve  an  evaluation  of  the 
contractor's  quality  procedures  and  verification  that 
the  contractor's  quality  practices  conform  to  those 
procedures.  [Ref.  9:p.  23]  The  reviews  also  evaluate 
the  Government's  in-plant  quality  assurance  program. 
Navy  activities  receive  copies  of  quality  system  reviews 
if  they  participate  in  the  review  with  DCAS  or  if  they 
request  a  copy.  [Ref.  10]  Copies  received  by  Navy 
activities  are  submitted  to  the  appropriate  Systems 
Command.  The  Systems  Commands  then  transmit  the  reviews 
to  NMQAO.  NAVSEA  activities  submit  copies  directly  to 
NMQAO.   [Ref.  7] 

10.  Certificates  of  Competency  (COCs) .  If  a  small  business 
is  determined  to  be  non-responsible  by  a  Government 
buying  activity,  the  small  business  can  request  that  the 
Small  Business  Administration  (SBA)  determine  whether 
the  business  is  responsible.  If  the  SBA  concludes  that 
the  small  business  is  responsible,  it  will  prepare  a  COC 
to  document  that  determination.  The  buying  activity 
must  then  treat  the  small  business  as  a  responsible 
offeror.  [Ref.  3:p.  1-8]  The  SBA  sends  hard  copies  of 
COCs  to  NMQAO  for  inclusion  in  PDREP  [Ref.  7]. 

The  CES  data  base  excludes: 

1.  Material  evaluations  for  base  application  and  local  use. 

2.  Contractors  developing  major  weapon  systems. 

3.  Medical  procurements,  material,  suppliers,  or 
evaluations. 

4.  Subsistence  procurements,  material,  suppliers,  or 
evaluations. 

5.  Unsatisfactory  material  condition  caused  by  improper 
handling  after  receipt,  deterioration  during  local 
storage,  or  inadequate  maintenance  or  operation. 
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6.  Transportation  discrepancies  caused  by  the  carrier. 

7.  Ammunition  and  explosives  accidents. 

8.  Nuclear  weapon  procurements,  material,  suppliers,  or 
evaluations. 

9.  Naval  Nuclear  Power  Plant  primary  system  procurements, 
material,  suppliers,  or  evaluations. 

10.   Strategic    Systems    Project    Office    procurements, 
,  suppliers,  or  material  evaluations.   [Ref.  3:p.  1-3] 

C.   RED  YELLOW  GREEN  PROCEDURES 

A  one  year  test  of  the  RYG  program  commenced  1  August 
1989.  It  is  being  conducted  at  the  following  five  Navy 
activities:  Naval  Air  Engineering  Center  Lakehurst,  Naval 
Avionics  Center  Indianapolis,  Naval  Ships  Parts  Control  Center 
(SPCC)  Mechanicsburg  (Code  021,  Level  1/SS) ,  Naval  Supply 
Center  Charleston/Naval  Shipyard  Charleston,  and  Naval  Supply 
Center  Pensacola/Naval  Aviation  Depot  Pensacola.   [Ref.  2] 

The  RYG  program  is  a  source  selection  tool  that  uses 
information  documented  in  the  CES  to  evaluate  contractor  past 
performance.  Based  on  information  in  the  CES,  contractors  are 
classified  by  FSC  as  high  risk  quality  performers  (Red) , 
moderate  risk  (Yellow) ,  or  low  risk  (Green) .  New  contractors 
or  those  for  which  the  CES  does  not  have  enough  data  are 
classified  as  "Insufficient  Data."  Because  the  classification 
is  by  FSC  and  contractor,  one  contractor  may  have  several 
color  classifications  if  he  supplies  the  Government  with  more 
than  one  commodity.   [Ref.  Irencl.  (1),  p.  3] 
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The  general  descriptions  of  the  color  classifications  as 
outlined  in  the  program  procedures  are: 

1.  Red.  The  contractor's  performance  history  indicates  he 
has  supplied  goods  or  services  in  a  particular  FSC  of 
poor  enough  quality  to  warrant  requirement  of  special 
quality  actions  and  higher  level  review  before  contract 
award.   He  is  designated  as  a  high  quality  risk. 

2.  Yellow.  The  contractor's  performance  history  indicates 
he  has  supplied  goods  or  services  in  a  particular  FSC  of 
poor  enough  quality  to  warrant  requirement  of  special 
quality  actions.  He  is  designated  as  a  moderate  quality 
risk. 

3.  Green.  The  contractor's  performance  history  indicates 
that  he  has  supplied  high  quality  goods  or  services  and, 
therefore,  is  designated  as  a  low  quality  risk.  His 
proposal  is  evaluated  without  anticipating  special 
quality  actions.   [Ref.  l:encl.  (1),  pp.  3-4] 

Contractors  are  classified  as  Red,  Yellow,  or  Green  based  on 

the  specific  criteria  listed  in  Appendix  A  [Ref.  l:encl.  (1), 

attachment  (2)  ]  .   The  RYG  data  base  will  be  updated  monthly  to 

reflect  a  particular  contractor's  current  classification  [Ref. 

l:encl.  (1) ,  p.  5]  . 

If  no  negative  quality  performance  data  are  received  on 

Red  or  Yellow  contractors  during  the  previous  12  months,  they 

will  be  reclassified.   Red  contractors  will  automatically  be 

reclassified  as  Yellow,  and  Yellow  contractors  will  be  placed 

in  the  Insufficient  Data  classification.   If  a  contractor  was 

classified  as  Red  or  Yellow  because  of  his  reject  rate  on 

MIRs,  and  if  no  rejects  have  occurred  on  the  inspection  of 

five  or  more  lots  of  material  during  the  past  six  months,  he 

will  be  reclassified  as  Yellow  or  Green  respectively.   If  a 

contracting   officer  has   reason   to   question  a   commodity 
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classification,  NMQAO  should  be  contacted  for  confirmation  of 
the  current  classification.   [Ref.  l:encl.  (1),  p.  5] 

Contractor  classifications  will  be  updated  by  NMQAO 
monthly  based  on  new  data  received.  After  each  update,  NMQAO 
will  send  a  computer  tape  to  each  of  the  five  test  activities 
reflecting  the  new  contractor  classifications.  The  most 
current  data  will  then  be  used  by  the  activities  in 
performance  of  the  test.   [Ref.  12] 

The  RYG  test  procedures  stress  that  a  contractor's  color 

classification  should  never  keep  him  from  competing  for  a 

Government  contract.   They  specifically  state: 

The  procedures  set  forth  herein  do  not  in  any  way  obliterate 
the  requirement  that  a  written  responsibility  determination 
is  to  be  made  for  every  requirement  prior  to  award.  A 
contractor's  color  classification  alone  is  not  sufficient  to 
determine  that  a  contractor  is  or  is  not  responsible  without 
further  consideration  of  the  standards  set  forth  in  FAR 
9.104.   [Ref.  l:encl.  (1),  p.  4] 

The  detailed  procedures  of  the  RYG  program  are  divided 
into  three  categories,  simplified  small  purchase,  major 
purchase,  and  fixed  price-greatest  value  procedures.  A 
description  of  the  application  and  specific  requirements  of 
each  of  these  categories  follows. 

1 .   Simplified  Small  Purchase  Procedures 

The  simplified  small  purchase  procedures  will  be 
tested  at  all  five  test  activities.  They  apply  to  all  oral  or 
written  quotations  solicited  during  the  one  year  test  for 
selected  FSCs  that  will  result  in  purchase  orders  with  an 
estimated  value  greater  than  $2500.   The  FSCs  used  for  the 
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test  were  chosen  by  each  test  activity.  They  may  be  modified 
during  the  test  by  agreement  between  the  test  activity  and 
NMQAO.  Each  activity  can  choose  whether  to  employ  these 
procedures  for  purchases  under  blanket  purchase  agreements, 
imprest  fund  purchases,  and  delivery  orders  against 
established  contracts  or  with  the  General  Services 
Administration.  However,  if  RYG  procedures  are  used  for  these 
purchases,  the  activity  must  issue  a  written  or  confirming 
purchase  order  to  document  the  purchase  for  the  file.  Waiver 
of  the  use  of  these  procedures  may  be  granted  by  the  Chief  of 
the  Contracting  Office  for  all  test  sites  except  SPCC.  At 
SPCC,  the  Director  of  the  Hull,  Mechanical,  and  Electrical 
Contracting  Department  must  grant  any  waivers.  [Ref.  Irencl. 
(1) ,  pp.  1-2] 

When  a  purchase  made  under  these  procedures  requires 
a  synopsis  in  the  Commerce  Business  Daily,  the  notice  will 
explain  that,  although  price  will  be  a  significant  factor  in 
proposal  evaluation,  other  factors,  including  the  contractor's 
quality  history,  will  also  influence  the  award  decision. 
Likewise,  the  solicitation  document  must  include  a  clause 
explaining  the  RYG  program  to  prospective  offerors  and 
advising  them  of  its  use  for  the  solicited  procurement.  [Ref. 
l:encl.  (1),  p.  6]  The  clauses  to  be  used  for  simplified 
small  purchase  procedures  are  provided  in  Appendix  B  [Ref. 
l:encl.  (1),  attachment  (6)]. 
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since  the  basis  of  a  contractor's  color  classification 
by  FSC  does  not  vary,  the  level  of  quality  risk  is  the  same 
regardless  of  the  procedures  used  for  the  procurement  (i.e., 
small  purchase,  major  procurement,  or  greatest  value) . 
However,  the  procedures  that  contracting  personnel  follow 
after  receiving  proposals  does  vary.  Under  the  simplified 
small  purchase  procedures,  the  following  procedures  are 
followed. 

First,  each  offeror's  classification  must  be 
determined  from  the  data  base — Red,  Yellow,  Green,  or 
Insufficient  Data.  If  the  contractor  is  classified  Green,  his 
proposal  is  evaluated  as  received  since  past  performance 
indicates  that  quality  should  not  be  a  concern  if  he  is 
selected.  If  the  offeror  is  classified  Yellow,  the 
appropriate  TEA  is  added  to  his  quoted  price.  The  adjusted 
price  should  better  represent  what  the  actual  cost  to  the 
Government  would  be  (considering  the  cost  to  correct  quality 
problems)  if  he  were  selected.  If  the  contractor  is 
classified  Red,  again  his  quoted  price  is  adjusted  by  adding 
an  appropriate  TEA.  This  TEA  will  be  higher  than  the  one 
assigned  to  a  Yellow  contractor  since  the  costs  the  Government 
incurs  while  doing  business  with  a  Red  contractor  will 
probably  be  greater  than  that  of  a  contractor  classified  as 
Yellow.  If  there  are  insufficient  data  to  assign  the 
contractor  a  color  classification  for  the  FSC  involved  in  the 
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solicitation,  there  is  no  basis  for  adding  a  TEA  to  his  cost 
proposal,  so  none  is  assigned.   [Ref.  l:encl.  (1),  pp.  3-4] 

After  TEAS  are  assigned  to  proposals  from  Red  and 
Yellow  contractors,  adjusted  prices  are  used  in  the  source 
selection  process.  If  a  contract  is  awarded  to  other  than  a 
Green  contractor,  additional  actions  must  be  taken  by  the 
purchasing  activity  during  the  test  period.  It  is  anticipated 
that  similar  actions  will  be  required  when  RYG  is  exported  for 
general  use  by  procuring  activities.  If  award  is  made  to  a 
Red  contractor,  for  example,  the  head  of  the  contracting 
office  must  approve  the  award.  The  RYG  test  procedures 
stress,  however,  that  under  no  circumstances  is  a  Red 
contractor  to  be  barred  from  competing  for  a  contract.  [Ref. 
l:encl.  (1),  p.  4]  If  award  is  made  to  either  a  Red  or  Yellow 
contractor,  contracting  personnel  should  ensure  that  any 
necessary  quality  control  requirements  are  included  in  the 
purchase  order  [Ref.  l:encl.  (1),  p.  6].  If  award  is  made  to 
a  contractor  on  which  the  Government  has  insufficient  quality 
data,  even  though  no  TEAs  were  assigned,  the  procuring 
activity  may  choose  to  require  greater  quality  controls  than 
they  would  if  dealing  with  a  Green  contractor  [Ref.  Irencl. 
(1) ,  p.  4]. 

The  TEAS  for  the  simplified  small  purchase  procedures 
are  set  at  standard  values  [Ref.  l:encl.  (1),  p.  2].  The  TEA 
for  a  Red  contractor  is  $2499  and  for  a  Yellow  contractor  is 
$1255  [Ref.  Irencl.  (1),  attachment  (4)].   These  values  are 
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based  on  the  anticipated  requirement  for  the  following 
additional  Government  quality  actions.  For  a  Red  contractor, 
additional  requirements  would  be  Government  source  inspection 
(GSI)  ,  receipt  inspection  at  destination  performed  with  a  Navy 
representative  present,  and  a  quality  assurance  letter  of 
instruction.  For  a  Yellow  contractor,  the  requirements  would 
be  GSI  and  a  quality  assurance  letter  of  instruction.  [Ref. 
Irencl.  (1),  attachment  (3)]  The  dollar  values  were 
calculated  by  estimating  the  costs  the  Government  would  incur 
in  performing  these  activities.  Specifically,  GSI  is  assigned 
a  cost  of  $500  by  estimating  this  would  take  the  Government  14 
hours  at  $34.18  per  hour.  Receipt  inspection  at  destination 
is  expected  to  cost  the  Government  $1194  based  on  eight  hours 
at  $43  per  hour  plus  $200  for  material  handling  and  $650  for 
testing.  Finally,  the  quality  assurance  letter  of  instruction 
is  estimated  to  cost  $755  based  on  eight  hours  at  $34.18  per 
hour  for  DCAS  personnel  plus  16  hours  at  $3  0  per  hour  for 
procurement  representative  personnel.  [Ref.  Irencl.  (2), 
attachment  (3)] 

2 .   Major  Purchase  Procedures  with  TEAs 

The  major  purchase  procedures  with  TEAs  will  be  used 
at  only  two  of  the  five  test  activities,  SPCC  Mechanicsburg 
(Code  021,  Level  I/SS)  and  Naval  Supply  Center  Pensacola/Naval 
Aviation  Depot  Pensacola.  These  procedures  apply  to  all 
negotiated  competitive  solicitations  with  an  estimated  value 
greater  than  $2500  for  the  FSCs  covered  by  the  test  during  the 
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one  year  test  period  when  large  purchase  procedures  are  used. 

As  with  the  simplified  small  purchase  procedures,  each  test 

activity  has  chosen  the  FSCs  to  be  analyzed  during  the  test, 

and  these  may  be  modified  by  agreement  between  the  test  site 

and  NMQAO.   Waiver  of  the  procedures  may  be  granted  by  the 

Chief  of  the  Contracting  Office  or,  at  SPCC,  by  the  Director 

of   the   Hull,   Mechanical,   and   Electrical   Contracting 

Department.   [Ref.  l:encl.  (2),  pp.  2-3] 

When  a  procurement  covered  by  these  test  procedures 

requires  a  synopsis  in  the  Commerce  Business  Daily,  the  notice 

will  explain  that,  although  price  will  be  a  significant  factor 

in  proposal  evaluation,  other  factors  as  detailed  in  the 

solicitation,  including  the  offeror's  quality  history,  will 

influence  the  award  decision  [Ref.  Irencl.  (2),  p.  7].   The 

solicitation  will  include  essentially  the  following  notice  to 

offerors: 

This  procurement  is  part  of  a  test  of  the  Navy's  Contractor 
Evaluation  System,  "Red/Yellow/Green"  Program.  Award  will 
be  based  upon  the  Contracting  Officer's  decision  as  to  which 
offer  provides  the  best  value  to  the  Navy — price,  past 
quality  performance,  and  other  factors  considered.  Details 
are  provided  in  the  provisions  entitled  "NOTICE  TO 
PROSPECTIVE  OFFERORS"  (Section  L)  and  "ADDITIONAL  EVALUATION 
FACTORS — TEST  OF  CONTRACTOR  EVALUATION  SYSTEM  (MAJOR 
PURCHASE  PROCEDURES)"  (Section  M).   [Ref.  l:encl.  (2),  p.  7] 

The  clauses  to  be  included  in  Sections  L  and  M  of  the 

solicitation  are  provided  in  Appendix  C  [Ref.  Irencl.   (2), 

attachment  (5) ] . 

When  proposals  are  received,  contracting  personnel 

query  the  RYG  data  base  to  determine  the  color  classification 
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of  each  offeror  for  all  FSCs  covered  by  the  solicitation.  As 
with  simplified  small  purchase  procedures,  a  Green 
classification  indicates  that  the  offeror  is  a  low  quality 
risk  and,  therefore,  no  TEAs  are  assigned  to  his  proposal.  If 
the  contractor  has  a  Yellow  classification,  the  procurement 
package  and  the  contractor's  quality  history  will  be  used  to 
determine  necessary  Government  quality  assurance  actions  and 
associated  TEAs.  The  TEAs  are  then  added  to  the  offeror's 
proposed  price.  If  the  contractor  has  a  Red  classification, 
appropriate  TEAs  are  determined  in  a  manner  similar  to  those 
for  Yellow  contractors.  However,  a  greater  number  of  quality 
assurance  actions  will  be  expected  when  dealing  with  a  Red 
contractor.  Therefore,  the  TEAs  assigned  will  probably  be 
higher.  If  the  contractor  does  not  have  a  color 
classification  because  of  insufficient  data,  he  will  not  be 
assigned  any  TEAs.  [Ref.  Irencl.  (2),  p.  4]  Additionally,  if 
the  procurement  is  from  a  sole  source  supplier,  no  TEAs  will 
be  assessed  regardless  of  the  applicable  color  classification 
[Ref.  Irencl.  (2),  p.  6]. 

The  TEAs  for  major  purchase  procedures  are  based  on 
the  specific  quality  deficiencies  in  the  performance  history 
of  the  contractor  for  a  particular  ESC.  These  deficiencies 
are  used  to  determine  which  additional  quality  assurance 
actions  the  Government  should  take  to  ensure  receipt  of  a 
quality   good   or    service.   [Ref.   Irencl.   (2),    p.   2] 
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Contracting  personnel  take  the  following  steps  when  assigning 
TEAS  under  major  purchase  procedures. 

First,  contracting  officers  ascertain  the  reason  for 
the  contractor's  classification  being  Red  or  Yellow  by  pulling 
his  classification  code  from  the  RYG  data  base  (listed  in 
Appendix  A)  .  Second,  they  decide  which  additional  quality 
assurance  actions  the  Government  must  take.  A  guideline  is 
provided  to  assist  them  (Appendix  D)  [Ref.  Irencl.  (2), 
attachments  (3)&(4)].  As  can  be  seen  in  Appendix  D,  the 
contractor's  deficiency  code  or  codes  are  tied  to  an  alpha- 
numeric code  for  additional  quality  assurance  requirements. 
Now,  contracting  personnel  perform  the  final  step,  assignment 
of  TEAS.  The  alpha-numeric  codes  for  quality  assurance 
actions  correspond  to  the  list  of  quality  assurance 
requirements  and  associated  TEAs  in  Appendix  D.  The  TEAs 
listed  are  calculated  in  the  same  way  as  those  used  in  the 
simplified  small  purchase  procedures.  Specific  calculations 
for  each  TEA  are  detailed  in  Appendix  D.  The  test  procedures 
stress,  however,  that  the  costs  on  which  the  listed  TEAs  are 
based  are  provided  as  examples.  Since  labor  rates  and  test 
costs  may  vary  considerably  between  procuring  activities,  each 
test  activity  should  recalculate  the  TEAs  to  reflect  the 
procuring  activity's  estimated  costs. 

After  TEAs  are  assigned  to  appropriate  proposals,  all 
proposals  are  evaluated  together  (using  adjusted  prices  for 
contractor's  assigned  a  TEA)  [Ref.  l:encl.  (2),  p.  6].   If  the 
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contract  is  awarded  to  a  Red  or  Yellow  contractor,   the 

contracting  officer  must  ensure  that  the  additional  quality 

assurance  actions  used  to  calculate  TEAs  are  called  out  as 

requirements  in  the  contract  [Ref.  l:encl.  (2),  p.  7].   If  the 

contract  is  awarded  to  a  Red  offeror,  approval  for  the  award 

must  be  granted  by  the  head  of  the  contracting  office  [Ref. 

Irencl.  (2),  p.  4].   If  a  contractor  on  which  the  Government 

has  insufficient  quality  data  wins  the  award,  the  procuring 

activity  will  require  appropriate  quality  assurance  actions  to 

ensure  a  certain  quality  level  even  though  TEAs  were  not 

assigned  [Ref.  l:encl.  (2),  p.  4]. 

3 .   Major  Purchase  Procedures  with  Fixed  Price-Greatest 
Value 

The  greatest  value  procedures  will  be  used  at  only 

three  of  the  five  test  activities;  Naval  Air  and  Engineering 

Center  Lakehurst,  Naval  Avionics  Center  Indianapolis,  and 

Naval  Supply  Center  Charleston/Naval  Shipyard  Charleston  [Ref. 

l:encl.  (3),  p.  2].   These  procedures  apply  to  all  negotiated 

competitive  solicitations  with  an  estimated  value  greater  than 

$2500  for  the  FSCs  covered  by  the  test  during  the  one  year 

test  period  where  sealed  bidding  procedures  would  otherwise  be 

most  appropriate  [Ref.  Irencl.  (3),  p.  2].   Each  test  activity 

has  chosen  the  FSCs  that  will  be  covered  by  the  test;  these 

may  be  modified  by  agreement  between  the  activity  and  NMQAO 

[Ref.  l;encl.  (3),  p.  3].   Waiver  of  the  procedures  may  be 
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granted  by  the  Chief  of  the  Contracting  Office  [Ref.  l:encl. 
(3) ,  p.  3]. 

Solicitations  prepared  under  these  procedures  will 
include  the  same  notice  used  under  the  major  purchase 
procedures  employing  TEAs  [Ref.  l:encl.  (3),  p.  7].  The 
clauses  to  be  included  in  Sections  L  and  M  of  the  solicitation 
are  provided  in  Appendix  E  [Ref.  Irencl.  (3),  attachment  (3)]. 
The  source  selection/ evaluation  plan  will  be  prepared  by  the 
Quality  staff  and  will  specify  the  criteria  to  be  used  in 
evaluating  Red,  Yellow,  and  Green  commodities.  This  plan  will 
be  forwarded  with  the  procurement  request  to  the  contracting 
officer  for  review  and  approval.  [Ref.  l:encl.  (3),  p.  3] 
For  those  commodities  covered  by  these  procedures,  the  source 
selection/ evaluation  plan  will  assign  price  a  minimum  weight 
of  40  percent,  with  the  remainder  assigned  to  quality 
assurance  considerations  under  RYG  procedures  [Ref.  Irencl. 
(3),  p.  2].  During  the  test,  no  factors  other  than  these  two 
shall  be  considered  without  prior  approval  from  Naval  Supply 
Systems  Command  (NAVSUP)  Code  02  via  NAVSUP  Code  021  [Ref. 
Irencl.  (3),  p.  2].  Appendix  F  [Ref.  Irencl.  (3),  p.  8] 
provides  an  example  of  the  RYG  evaluation  portion  of  the 
source  select ion/ evaluation  method. 

If  a  contractor  is  determined  to  be  Green,  he  will 
normally  be  rated  as  "Excellent"  or  "Good."  A  Yellow 
classification  will  normally  carry  with  it  a  rating  of  "Good," 
"Average,"  or  "Marginal."   Red  commodities  will  usually  be 
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rated  "Marginal"  or  "Unacceptable."  If  there  are  insufficient 
data  to  evaluate  the  offeror,  the  proposal  will  be  considered 
technically  acceptable  and  will  be  evaluated  solely  on  the 
basis  of  price.  The  contracting  officer  can  exercise 
discretion  in  the  use  of  adjectives  (i.e.,  excellent,  good, 
etc.)  for  each  color  classification  or  other  locally 
determined  source  selection  criteria.  [Ref.  l:encl.  (3),  pp. 
4-5] 

When  proposals  are  received,  the  contracting  office 
will  forward  the  names  and  Commercial  and  Government  Entity 
(CAGE)  codes  of  all  offerors  to  the  Quality  staff.  The 
Quality  staff  will  determine  the  color  classification  and  the 
associated  reason  codes  for  each  offeror  for  each  FSC.  They 
will  then  assign  ratings  to  each  offeror  based  on  the  criteria 
outlined  in  the  source  selection  plan.  If  commodities  of 
different  offerors  lie  within  the  same  color  classification, 
their  relative  differences  will  be  explained  in  an 
accompanying  narrative.  This  evaluation  will  be  forwarded  to 
the  contracting  office.  [Ref.  Irencl.  (3),  p.  3] 

If  the  contract  is  awarded  to  a  Red  or  Yellow 
contractor,  the  Quality  staff  will  determine  any  necessary 
additional  quality  assurance  requirements  that  should  be 
imposed  on  the  contractor.  The  contracting  officer  will 
ensure  that  these  are  included  in  the  contract.  [Ref.  Irencl. 
(3),  p.  3]   If  the  contract  is   awarded  to   a  Red  contractor. 
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approval  for  the  award  must  be  granted  by  the  head  of  the 
contracting  office  [Ref.  Irencl.  (3),  p.  3]. 

D.   SUMMARY 

This  chapter  described  the  Navy's  RYG  program.  First,  it 
explained  reasons  behind  establishment  of  the  program. 
Second,  it  described  the  CES  data  base  on  which  the  program  is 
based.  Finally,  it  detailed  the  test  procedures  that  are 
being  used  to  implement  the  program.  The  next  chapter  will 
briefly  describe  three  other  current  DOD  contractor  quality 
programs. 
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III.   OTHER  DEPARTMENT  OF  DEFENSE  QUALITY  INITIATIVES 

A.  INTRODUCTION 

One  factor  in  the  researcher's  evaluation  of  the  Red 
Yellow  Green  (RYG)  program  is  an  assessment  of  how  this 
program  compares  with  other  contractor  quality  programs  being 
used  or  considered  for  use  by  the  Navy.  Three  programs 
evaluated  by  the  researcher  are  the  Air  Force  Contractor 
Performance  Assessment  Reporting  System  (CPARS) ,  the  Naval 
Avionics  Center  (NAC)  Indianapolis  Blue  Ribbon  Contractor 
program,  and  the  Department  of  Defense  (DOD)  Total  Quality 
Management  (TQM)  initiative.  These  three  programs  were  chosen 
for  the  assessment  because  each  has  been  considered  for  use 
by  the  Navy  as  an  alternative  to  RYG.  Therefore,  a  discussion 
of  the  merits  and  shortcomings  of  these  programs  and  their 
relationship  to  RYG  is  considered  important.  The  evaluation 
described  above  will  be  the  subject  of  Chapter  IV  of  this 
thesis.  In  preparation  for  the  assessment  in  Chapter  IV,  this 
chapter  provides  background  information  on  the  procedures 
associated  with  each  of  the  three  programs. 

B.  CONTRACTOR  PERFORMANCE  ASSESSMENT  REPORTING  SYSTEM 

The  first  program  evaluated  is  CPARS  used  by  the  Air 
Force.  As  stated  in  Air  Force  Systems  Command  (AFSC) 
Regulation  800-54,  "The  sole  purpose  of  CPARS  is  to  provide 
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for  a  command-wide  performance  data  base  used  in  AFSC  source 
selections."  [Ref.  13]  CPARS  currently  includes  84 
contractors  and  their  divisions  and  subsidiaries.  These 
contractors  were  chosen  because  the  Air  Force  conducts  most  of 
its  business  with  them.  Therefore,  it  was  determined  that  the 
benefits  of  maintaining  performance  data  on  them  would  be 
worth  the  effort  involved  in  the  CPARS  process.  [Ref.  14]  A 
Contractor  Performance  Assessment  Report  (CPAR)  is  prepared 
on  contracts  with  the  84  contractors  included  in  the  program 
valued  at  more  than  $5  million.  Preparation  of  the  reports 
are  further  limited  to  contracts  for  concept  demonstration  and 
validation,  full-scale  development,  and  full-rate  production 
and  deployment.  The  program  excludes  contracts  for  laboratory 
work,  services,  and  operations  and  maintenance.  [Ref.  13 :p. 
1]  Following  are  the  areas  of  contractor  performance 
evaluated  in  the  CPAR: 

1.  Product  system  performance. 

a.  Engineering  design/support; 

b.  Software  development. 

2.  Schedule. 

3.  Cost  performance. 

4.  Product  assurance. 

5.  Test  and  evaluation. 

6.  Integrated  logistics  support  program. 

7.  Management  responsiveness. 

8.  Subcontract  management. 
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9.   Variances. 

a .  Cost ; 

b.  Schedule; 

10.  Other  areas  unique  to  the  contract  evaluated.  [Ref.  15] 
The  CPAR  is  prepared  by  the  program  director  or  program 
manager.  An  initial  CPAR  for  a  new  contract  is  prepared 
between  six  months  and  one  year  after  contract  award. 
Following  this,  CPARs  are  submitted  annually  during  the 
contract  performance  period.  Additional  CPARs  are  completed 
when  the  program  manager  changes  or  if  the  program  manager 
determines  that  a  contractor's  performance  has  changed 
sufficiently  to  warrant  modification  of  the  most  recent  CPAR. 
In  addition,  the  contractor  can  request  that  a  CPAR  be 
completed  before  the  annual  report  is  prepared  if  he  believes 
that  his  performance  has  changed  enough  to  alter  the 
assessment.  The  final  CPAR  is  prepared  within  six  months 
after  the  final  major  end  item  is  delivered  or  when  a  contract 
is  terminated.   [Ref.  13 :p.  2] 

Preparation  of  the  CPAR  involves  four  steps.  First,  the 
program  manager  or  engineer  responsible  for  the  contract 
prepares  a  preliminary  assessment  with  associated 
documentation.  Second,  the  program  manager  forwards  the 
original  of  the  preliminary  CPAR  to  the  contractor  and  retains 
a  copy.  The  contractor  has  3  0  days  to  respond  to  the  CPAR. 
Upon  receiving  a  response  from  the  contractor  or  3  0  days  after 
transmitting  the  CPAR  to  the  contractor,  whichever  occurs 
first,  the  program  manager  takes  the  third  step  and  signs  the 
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CPAR.   He  may  first  revise  his  evaluation  if  the  contractor's 

response  indicates  that  a  change  is  appropriate.   Finally,  the 

program  manager  sends  the  signed  CPAR  with  its  attachments  to 

the  Product  Division  of  the  AFSC  having  cognizance  over  his 

program  for  input  into  the  command-wide  data  base.    [Ref. 

13:p.  2] 

As  part  of  his  assessment,  the  program  manager  translates 

his   evaluation   of   the   contractor   in   each   of   the   ten 

performance  evaluation  areas  into  a  color  classification.  The 

following  is  a  description  of  the  criteria  he  must  use  in 

assigning  colors: 

Blue  (Exceptional) .  Indicates  performance  clearly 
exceeds  contractual  requirements.  The  area  of  evaluation 
contains  few  minor  problems  for  which  corrective  actions 
appear  highly  effective.  For  cost  performance,  blue 
indicates  a  positive  cost  variance. 

Green  (Satisfactory) .  Indicates  performance  clearly 
meets  contractual  requirements.  The  area  of  evaluation 
contains  some  minor  problems  for  which  corrective  actions 
appear  satisfactory.  For  cost  performance,  green  indicates 
no  cost  variance  or  a  negative  cost  variance  greater  than 
zero  but  less  than  or  equal  to  five  percent. 

Yellow  (Marginal) .  Indicates  performance  meets 
contractual  requirements.  The  area  of  evaluation  contains 
a  serious  problem  for  which  corrective  actions  have  not  yet 
been  identified,  appear  only  marginally  effective,  or  have 
not  been  fully  implemented.  For  cost  performance,  yellow 
indicates  a  negative  cost  variance  greater  than  five  percent 
but  less  than  or  equal  to  15  percent. 

Red  (Unsatisfactory) .  Indicates  the  contractor  is  in 
danger  of  not  being  able  to  satisfy  contractual  requirements 
and  recovery  is  not  likely  in  a  timely  manner.  The  area  of 
evaluation  contains  serious  problems  for  which  corrective 
actions  appear  ineffective.  For  cost  performance,  red 
indicates  a  negative  cost  variance  greater  than  15 
percent.   [Ref.  13:p.  7] 
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since  each  area  is  evaluated  by  color,  a  contractor  might 
receive  several  color  classifications  under  one  contract.  For 
example,  he  might  be  classified  Red  in  quality.  Green  in  cost 
control,  and  Blue  in  the  remainder. 

When  making  assessments  of  contractors,  the  AFSC 
Regulation  covering  CPARS  cautions  program  managers  to  base 
their  evaluations  on  facts.  Their  assessments  should  be 
supported  by  contract  management  data  "such  as  cost 
performance  reports,  technical  interchange  meetings,  financial 
solvency  assessments,  production  management  reviews, 
contractor  operations  reviews,  functional  performance 
evaluations,  and  earned  contract  incentives."  [Ref.  13 :p.  1] 
The  Regulation  specifically  states,  "Subjective  assessments 
concerning  the  causes  or  ramifications  of  the  contractor's 
performance  should  be  provided;  however,  speculation  or 
conjecture  should  not  be  included."   [Ref.  13: p.  1] 

The  intention  of  the  program  is  that  during  source 
evaluation,  CPARs  will  be  used  to  evaluate  a  contractor's  past 
performance  and  assess  the  risk  involved  in  doing  business  for 
the  contract  under  consideration.  Specifically,  during  the 
source  selection  process,  the  contracting  officer  will  check 
the  CPARs  of  contractors  submitting  proposals.  If  he 
determines  that  the  effort  described  in  the  CPARs  applies  to 
the  current  procurement,  they  will  be  assembled  into  a  report 
for  the  Performance  Risk  Analysis  Group  (PRAG) .  The  PRAG 
considers   the   CPARs   along   with   other   contractor   past 
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performance  data  to  determine  contractor  risk.  The  PRAG 
usually  expresses  contractor  risk  as  low,  medium,  or  high. 
This  risk  assessment  is  then  used  as  part  of  the  source 
selection.   [Ref.  14] 

At  present,  each  Product  Division  within  the  AFSC 
maintains  a  complete  set  of  hard  copy  CPARs.  The  Air  Force 
intends  eventually  to  automate  CPARS.  The  automation  effort 
has  been  delayed  for  two  reasons.  First,  the  resources 
required  to  input  the  CPARs  and  ensure  compatibility  with  the 
computer  hardware  at  all  commands  are  not  currently  available. 
Second,  the  Air  Force  is  still  developing  a  method  to  ensure 
the  security  of  the  automated  data.   [Ref.  14] 

Each  CPAR  will  be  retained  for  five  years  unless  the 
program  manager  requests  a  longer  retention  period.  CPARs  are 
considered  source  selection  sensitive  because  they  will  be 
used  for  procurements  throughout  their  retention  period.  They 
may  also  contain  proprietary  information.  Therefore,  all 
CPARs  and  attachments  will  be  marked  "For  Official  Use  Only/ 
Source  Selection  Sensitive."  Additionally,  the  forms  and  the 
data  base  (if  automated)  must  be  protected  from  disclosure  to 
organizations  or  personnel  not  involved  in  the  source 
selection  process.   [Ref.  13:p.  3] 

C.   BLUE  RIBBON  CONTRACTOR  PROGRAM 

The  Blue  Ribbon  Contractor  program  at  NAC  Indianapolis 
began  in  December  1987.    It  was  inspired  by  the  Packard 
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Commission  recommendations  of  1986  [Ref.  16]  and  the  Blue 
Ribbon  Contractor  program  at  Tinker  Air  Force  Base  begun  a 
year  earlier  [Ref.  17].  NAC  has  attempted  to  learn  from  some 
of  the  difficulties  experienced  at  Tinker  and  has  tailored  the 
program  to  respond  to  their  concerns  and  conform  to  their 
resource  constraints  [Ref.  18]. 

The  program  at  NAC  is  applied  to  contracts,  purchase 
orders,  and  blanket  purchase  agreements  with  up  to  a  $25,000 
estimated  value  [Ref.  16:p.  41].  Procurements  made  for  the 
bomb-rack  program,  which  have  historically  caused  difficulties 
for  NAC,  are  excepted  from  this  dollar  limit  and  are  all 
included  in  the  Blue  Ribbon  Contractor  program  [Ref.  19]. 
Other  exemptions  to  the  program  are  procurements  awarded  via 
sealed  bidding  procedures  and  involving  small,  disadvantaged 
business  set-aside  awards.  Additionally,  the  program  is  not 
a  substitute  for  procedures  governing  contractor  respon- 
sibility determination  [Ref.  19:p.  2]. 

In  order  to  qualify  as  a  Blue  Ribbon  contractor,  a 
contractor  must  first  apply  in  writing  to  NAC.  If  he  has  made 
one  delivery  during  the  last  12  months,  he  will  be  placed  on 
the  list  in  that  category.  To  receive  the  benefits  of  being 
designated  a  Blue  Ribbon  contractor,  however,  he  must  have  and 
maintain  an  on-time  delivery  rate  of  95  percent  or  greater. 
[Ref.  19 :p.  2]  This  delivery  rate  is  calculated  by  dividing 
the  number  of  delinquent  orders  by  the  number  of  open  orders 
and  multiplying  by  100  [Ref.  18].   Only  data  collected  at  NAC 
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Indianapolis  are  used  in  determining  this  delivery  rate  [Ref. 
19:p.  3].  If  a  Blue  Ribbon  contractor  does  not  qualify  to 
receive  the  benefits  of  Blue  Ribbon  status  upon  application, 
or  if  his  on-tiitie  delivery  rate  drops  below  95  percent,  he  is 
notified  that  he  is  still  listed  as  a  Blue  Ribbon  contractor, 
but  he  cannot  exercise  the  benefits  of  the  program  until  his 
delivery  rate  improves  [Ref.  18]. 

There  are  three  benefits  accruing  to  Blue  Ribbon 
contractors  with  a  qualifying  delivery  rate.  First,  the 
contracting  officer  has  the  authority  to  award  to  a  Blue 
Ribbon  contractor  over  other  offerors  at  a  price  up  to  ten 
percent  higher  than  the  lowest  priced  responsible  offeror.  If 
the  procurement  is  in  support  of  the  Navy  bomb  rack  program, 
the  contracting  officer  has  the  authority  to  award  to  a  Blue 
Ribbon  contractor  at  a  price  up  to  2  0  percent  higher  than  the 
lowest  priced  responsible  offeror.  [Ref.  19 :p.  1]  Second, 
NAC  Indianapolis  solicits  Blue  Ribbon  contractors  to  the 
maximum  extent  possible.  The  bidders  list  is  rotated,  but  70 
percent  of  it  is  made  up  of  Blue  Ribbon  contractors.  [Ref. 
18]  Third,  Blue  Ribbon  contractors  are  permitted  to  use  their 
status  in  their  marketing  programs  [Ref.  18]. 

So  far,  the  Blue  Ribbon  Contractor  program  has  resulted  in 
only  3.5  percent  of  awards  (of  contracts  in  the  program)  being 
made  to  Blue  Ribbon  contractors  that  were  not  the  lowest 
priced  responsible  offerors.  However,  the  delinquency  rate  on 
blanket  purchase  agreements  has  dropped  from  67  percent  before 
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the  start  of  the  program  to  8 . 7  percent  in  July  1989.  Seventy 
percent  of  the  suppliers  awarded  contracts  in  July  1989  are 
the  same  suppliers  that  were  receiving  awards  when  the  program 
began.  One  protest  has  been  filed  by  a  low  offerer  who  lost 
a  contract  to  a  higher  priced  Blue  Ribbon  contractor.  The 
General  Accounting  Office  dismissed  the  protest  because  it  was 
not  timely.  [Ref.  18]  Therefore,  no  protest  related  to  the 
Blue  Ribbon  Contractor  program  has  yet  tested  the  program's 
criteria. 

D.   TOTAL  QUALITY  MANAGEMENT 

TQM  is  a  management  style  introduced  to  the  Japanese  in 
the  late  1940s  by  Americans  participating  in  the  post-World 
War  II  effort  to  rebuild  Japan.  TQM  proved  to  be  a  very 
successful  management  method  in  Japan.  As  a  result,  it  is 
gaining  an  increasing  following  within  U.S.  industry  trying 
to  regain  competitive  position  both  internationally  and 
domestically.  Based  on  successes  in  Japan  and  within  the 
U.S.,  DOD  is  attempting  to  adopt  TQM  in  an  effort  to  operate 
more  efficiently  in  a  time  of  dwindling  resources.  This 
section  of  Chapter  III  will  attempt  to  describe  the  TQM  effort 
as  it  applies  to  DOD  and,  more  specifically,  DOD  procurement. 
First,  it  will  describe  TQM  in  general,  concentrating  the 
discussion  on  those  aspects  of  TQM  that  most  directly  impact 
upon  procurement  practices  and,  therefore,  that  would  compete 
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with  RYG.   Second,  it  will  outline  current  DOD  implementation 
plans  related  to  acquisition. 

1.   TQM  Implementation — General 

Adopting  TQM  means  that  everyone  within  an 
organization  must  be  involved  in  improving  the  quality  of 
their  product.  Whether  they  are  manufacturing  a  product  or 
preparing  documents,  everyone  produces  something  and  every 
product  has  a  customer.  Therefore,  everyone  in  the 
organization  can  improve  quality.  The  basic  theory  behind  TQM 
is  that  as  quality  improves,  productivity  improves  because 
rework  is  reduced  [Ref .  20]  .  As  quality  and  productivity 
rise,  competitive  position  also  improves  [Ref.  21] . 

One  of  the  key  figures  in  the  introduction  of  TQM  to 
the  Japanese  is  Dr.  W.  Edwards  Deming.  Dr.  Deming  has  devised 
14  principles  for  management  which  he  believes  U.S.  firms  must 
follow  if  they  desire  to  adopt  TQM.   His  14  principles  are: 

1.  Create  constancy  of  purpose  toward  improvement  of 
product  and  service,  with  the  aim  to  become  competitive 
and  stay  in  business,  and  to  provide  jobs. 

2.  Adopt  the  new  philosophy.  We  are  in  a  new  economic  age. 
Western  management  must  awaken  to  the  challenge,  must 
learn  their  responsibilities,  and  take  on  leadership  for 
change. 

3.  Cease  dependence  on  mass  inspection  to  achieve  quality. 
Eliminate  the  need  for  inspection  on  a  mass  basis  by 
building  quality  into  the  product  in  the  first  place. 

4.  End  the  practice  of  awarding  business  on  the  basis  of 
price  tag.  Instead,  minimize  total  cost.  Move  toward 
a  single  supplier  for  any  one  item,  on  a  long-term 
relationship  of  loyalty  and  trust. 
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5.  Improve  constantly  and  forever  the  system  of  production 
and  service,  to  improve  quality  and  productivity,  and 
thus  constantly  decrease  costs. 

6.  Institute  training  on  the  job. 

7.  Institute  leadership  (see  Point  12).  The  aim  of  super- 
vision should  be  to  help  people  and  machines  and  gadgets 
to  do  a  better  job.  Supervision  of  management  is  in 
need  of  overhaul,  as  well  as  supervision  of  production 
workers. 

8.  Drive  out  fear,  so  that  everyone  may  work  effectively 
for  the  company. 

9.  Break  down  barriers  between  departments.  People  in 
research,  design,  sales,  and  production  must  work  as  a 
team,  to  foresee  problems  of  production  and  in  use  that 
may  be  encountered  with  the  product  or  service. 

10.  Eliminate  slogans,  exhortations,  and  targets  for  the 
work  force  asking  for  zero  defects  and  new  levels  of 
productivity.  Such  exhortations  only  create  adversarial 
relationships,  as  the  bulk  of  the  causes  of  low  quality 
and  low  productivity  belong  to  the  system  and  thus  lie 
beyond  the  power  of  the  work  force. 

11.  Eliminate  work  standards  (quotas)  on  the  factory  floor. 
Substitute  leadership.  Eliminate  management  by 
objective.  Eliminate  management  by  numbers,  numerical 
goals.   Substitute  leadership. 

12 .  Remove  barriers  that  rob  the  hourly  worker  of  his  right 
to  pride  of  workmanship.  The  responsibility  of  super 
visors  must  be  changed  from  sheer  numbers  to  quality. 
Remove  barriers  that  rob  people  in  management  and  in 
engineering  of  their  right  to  pride  of  workmanship. 
This  means,  inter  alia,  abolishment  of  the  annual  or 
merit  rating  and  of  management  by  objective. 

13.  Institute  a  vigorous  program  of  education  and  self- 
improvement. 

14.  Put  everybody  in  the  company  to  work  to  accomplish  the 
transformation.  The  transformation  is  everybody's  job. 
[Ref.  20:pp.  23-24] 

Although  all  14  points  are  interrelated,  two  have  the 

greatest  implications  for  changing  traditional  procurement 
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practices.  These  are  number  3,  "Cease  dependence  on  mass 
inspection"  and  number  4,  "End  the  practice  of  awarding 
business  on  the  basis  of  price  tag."  Therefore,  the 
discussion  in  this  section  will  focus  on  how  these  two 
principles  should  be  implemented. 

The  push  to  do  away  with  mass  inspection  is  based  on 
the  theory  that  inspection  is  ineffective  in  ensuring  quality. 
This  is  true  for  two  reasons.  First,  inspection  occurs  too 
late  in  the  process.  If  quality  has  not  been  built  in  during 
production,  it  cannot  be  inspected  in  [Ref.  20:pp.  28-29]. 
Inspection  can  only  separate  good  items  from  bad.  Second, 
even  if  100  percent  inspection  is  employed  to  separate  good 
from  bad,  some  defective  material  will  slip  through.  One 
hundred  percent  inspection  is  unreliable  because  inspectors 
are  human  and  make  errors,  no  matter  how  conscientious, 
because  of  boredom  and  fatigue.  [Ref.  20:p.  29]  In  addition 
to  being  ineffective,  mass  inspection  is  very  expensive. 
First,  it  is  very  labor  intensive,  so  it  requires  the 
employment  of  many  inspectors.  Second,  the  items  that  fail 
inspection  are  either  discarded  or  reworked.  The  first 
alternative  wastes  the  costs  of  material  and  production  used 
to  make  the  item.  The  second  alternative  wastes  the 
additional  material  and  man  hours  employed  in  reworking  the 
item.  These  are  all  resources  that  could  be  better  employed 
elsewhere. 
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In  place  of  mass  inspection,  Statistical  Process 
Control  (SPC)  should  be  employed.  SPC  is  a  method  to  build 
quality  into  a  product  during  production.  It  is  a  tool  to  be 
used  by  management  to  identify  the  reasons  a  product  is  not 
produced  with  consistency.  Management  charts  a  process, 
perhaps  one  segment  of  production,  to  see  the  fluctuations  in 
the  output  of  the  process.  If  the  process  exhibits  more  than 
simple  random  variation,  it  is  not  in  statistical  control. 
This  means  that  local  sources  of  trouble  are  affecting  the 
process.  [Ref.  21:p.  5]  Examples  of  local  problems  are 
differences  in  materials  used  in  production,  readjustment  of 
machinery  at  the  beginning  of  each  shift,  and  use  of  different 
procedures  on  each  shift.  These  local  problems  must  be 
eliminated  to  bring  the  process  into  statistical  control. 
Only  when  the  process  is  in  control,  can  improvements  be  made 
effectively.  This  is  because  if  the  process  is  not  in 
control,  management  will  not  be  able  to  observe  the  affects  of 
changes  made  to  it.  [Ref.  21  :p.  5]  Once  the  process  is  in 
control,  changes  made  to  it  will  shift  the  entire  control 
chart  in  either  a  favorable  or  unfavorable  direction,  giving 
management  feedback  about  its  innovations.  Management  can  now 
follow  the  Plan,  Do,  Check,  Act  (PDCA)  cycle.  First,  data  are 
analyzed  to  determine  what  change  might  improve  the  process. 
Second,  management  tests  the  idea  by  putting  it  into  practice. 
Third,  the  control  charts  are  monitored  to  determine  the 
affect  (favorable  or  unfavorable)  of  the  change.   Finally, 
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based  on  the  feedback  from  the  third  step,  the  change  is 
either  implemented  permanently  or  discarded.  The  PDCA  cycle 
is  then  begun  again.  The  result  is  continuous  improvement  of 
the  process  in  smaller  and  smaller  increments. 

The  key  to  employing  sampling  inspections  and  SPC  in 
place  of  mass  inspection  is  the  determination  of  which 
characteristics  to  monitor  and  which  inspection  method  to  use. 
Dr.  Ishikawa  explains  that  it  is  important  to  inspect  quality 
characteristics  that  are  unstable,  thus  causing  defects,  as 
well  as  vital  characteristics  that  affect  the  life  of  a 
product  [Ref.  22].  These  inspections  are  not  performed  after 
the  product  is  complete;  they  are  conducted  by  the  worker 
during  the  production  process.  In  addition,  only  a  sample  of 
the  items  produced  are  inspected  to  either  determine  whether 
the  process  is  in  control  or  to  be  sure  that  the  process 
remains  in  control.  Because  only  a  small  sample  of  items  are 
inspected,  inspectors  can  do  the  job  properly.  [Ref.  20: pp. 
29-30]  Dr.  Ishikawa  further  explains  that  the  goal  of  SPC 
should  be  a  product  with  100  per  cent  acceptable  quality.  SPC 
makes  this  goal  feasible.  [Ref.  22:pp.  117-118]  Mass 
inspection,  on  the  other  hand,  is  guaranteed  to  result  in  some 
percentage  of  failures. 

The  second  principle  that  applies  directly  to 
procurement  is  number  4,  "End  the  practice  of  awarding 
business  on  the  basis  of  price  tag."  What  this  principle 
means  is  that  when  purchasing  material,  an  organization  should 
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consider  the  lowest  total  cost  of  the  item  rather  than  the 

lowest  initial  cost  [Ref.  23].   The  organization  must  have 

long  term  goals.    Buyers  have  a  new  job.    They  have 

traditionally  sought  lower  and  lower  prices  for  material  with 

little  or  no  concern  for  quality  or  service.   [Ref.  21:p.  25] 

It  is  difficult  to  select  a  contractor  with  a  higher  initial 

price  since  it  can  be  very  subjective  when  defending  the 

selection   during   an   award   protest.     Now   buyers   must 

concentrate  on  quality  because,  as  Dr.  Deming  states: 

Price  has  no  meaning  without  a  measure  of  the  quality  being 
purchased.  Without  adequate  measures  of  quality,  business 
drifts  to  the  lowest  bidder,  low  quality  and  high  cost  being 
the  inevitable  result.  American  industry  and  the  U.S. 
Government,  civil  and  military,  are  being  rooked  by  rules 
that  award  business  to  the  lowest  bidder.   [Ref.  23 :p.  23] 

In  order  to  ensure  receipt  of  quality  goods  and 
services,  an  organization  must  do  two  things.  First,  it  must 
require  that  its  suppliers  practice  TQM.  Second,  it  must 
reduce  the  number  of  suppliers  it  deals  with  and  seek  long- 
term  relationships  with  these  suppliers. 

To  verify  that  a  supplier  practices  TQM,  an 
organization  should  require  the  supplier  to  provide 
statistical  evidence  of  quality.  This  means  supplying  the 
buying  organization  with  control  charts  and  proof  that  the 
supplier  is  working  on  changing  its  management  style  to 
conform  with  Deming's  principles.  [Ref.  21:p.  113]  The 
buying  organization  must  be  sure  that  the  control  charts 
provided  represent  the  proper  inspection  technique  taken  at 
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the  critical  points  in  the  production  process  using  the 
correct  sample  size.  This  requires  a  close  relationship 
between  the  buying  activity  and  the  company  supplying  the 
material.  The  activity  buying  material  must  be  sure  that  the 
supplier  is  capable  of  producing  a  quality  product  before  the 
item  is  produced.  In  Dr.  Deming's  words,  "The  customer  that 
waits  for  delivery  of  material  to  learn  what  he  has  bought 
will  take  what  he  gets."  [Ref.  20:pp.  40-41]  This  concept 
opens  a  new  world  of  responsibilities  to  the  contract 
administration  activities  within  DOD.  It  suggests  that  their 
traditional  role  inspecting  the  quality  of  material  after  it 
has  been  produced  must  change.  Instead,  they  would  be 
expected  to  participate  with  the  contractor  in  verifying  the 
quality  of  his  production  processes  before  and  during 
production  of  required  material. 

The  concentration  on  long-term  goals  and  close  buyer- 
supplier  relationship  requires  a  long-term  relationship  with 
a  small  number  of  suppliers.  This  is  in  direct  conflict  with 
the  Competition  in  Contracting  Act  of  1984  and  a  concept  that 
DOD  must  deal  with  to  successfully  implement  a  TQM  philosophy. 
The  buying  activity's  goal  is  still  to  purchase  material  of 
high  quality  at  the  least  cost.  However,  the  buying  activity 
must  also  devote  resources  to  assist  the  vendor  in  employing 
TQM  if  it  is  to  succeed.  Therefore,  as  the  number  of 
suppliers  an  activity  deals  with  increases,  the  number  of 
resources  devoted  to  ensuring  a  TQM  philosophy  also  increases. 
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Even  after  the  activity  is  working  with  suppliers  dedicated  to 
TQM  and  is  receiving  high  quality  material,  if  this  material 
is  supplied  by  two  different  vendors,  there  will  be 
differences.  These  differences  will  surface  when  the  material 
is  placed  into  the  production  process.  [Ref.  20:p.  35]  For 
these  reasons.  Dr.  Deming  encourages  organizations  to  reduce 
their  suppliers  for  a  given  end  item  to  one  [Ref.  20,  p.  23]. 
Ishikawa  believes,  however,  that  an  organization  should 
maintain  at  least  two  suppliers  for  a  given  product  in  case 
one  firm  is  unable  to  supply  the  item  [Ref.  24]. 

A  supplier  needs  the  promise  of  a  long-term  contract 
to  make  it  cost  effective  for  him  to  institute  TQM  and 
concentrate  on  continuously  improving  his  production  process 
[Ref.  21:p.  115].  Without  some  guarantee  of  continued 
business,  he  will  not  be  willing  to  invest  the  resources 
necessary  to  bring  his  processes  into  control  and  then  improve 
upon  them.  DOD  must  seriously  consider  the  awarding  of  multi- 
year  contracts  (two  to  five  years)  or  contracts  with  optional 
years  to  assure  a  contractor  that  DOD  is  supporting  a  basic 
TQM  mandate.  Contractors  are  currently  forced  to  concentrate 
on  making  a  profit  in  the  short  term  since  DOD  is  focused  on 
the  short  term.  If,  however,  a  long-term  relationship  can  be 
developed,  both  customer  and  supplier  will  benefit.  The 
customer  will  receive  high  quality  material  at  lower  costs. 
The  supplier  will  no  longer  need  to  employ  resources  in 
storing  and  reworking  defective  material.   [Ref.  21:p.  115] 
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2 .   TOM  Implementation — POD 

The  application  of  TQM  within  the  Federal  Government 
began  in  1986  when  President  Reagan  signed  Executive  Order 
12552.  The  goal  of  this  Executive  Order  is  to  significantly 
improve  the  productivity  of  the  Federal  Government  by  1992 
through  employment  of  TQM.  [Ref.  25]  In  1988,  DOD  initiated 
its  TQM  program,  issuing  the  Total  Quality  Management  Master 
Plan  in  August  of  that  year. 

The  strategy  of  the  Master  Plan  "aims  at  achieving  one 
broad,  unending  objective:  continuous  improvement  of  products 
and  services."  [Ref.  26]  In  order  to  achieve  this  objective, 
the  plan  includes  goals  for  the  short  term  (one  year)  ,  mid 
term  (three  years) ,  and  long  term  (seven  years) .  These  goals 
cover  a  broad  range  of  DOD  tasks.  They  state,  however,  that 
the  acquisition  community  will  be  the  first  to  implement  TQM 
and  that  others  will  follow  in  approximately  three  years  [Ref. 
26:p.  3].  Since  the  acquisition  community  is  leading  TQM 
implementation  and  this  thesis  will  concentrate  its  discussion 
of  TQM  in  the  areas  relating  to  RYG,  discussion  of  the  Master 
Plan  will  be  limited  to  procurement  and  Government-industry 
relationships . 

In  the  short  term,  two  of  the  seven  goals  directly 
impact  procurement.  The  first  of  the  two  is  to  "Begin 
enlisting  Defense  industry  commitment."  This  goal  intends  to 
begin  TQM  implementation  by  using  programs  already  in 
existence  within  DOD.    These  programs  are  to  be  used  to 
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communicate  to  industry  DOD's  desire  for  continuous  process 
improvement.  [Ref.  26 :p.  10]  The  second  applicable  goal  is 
to  "Develop  and  implement  recognition  and  reward  system  based 
on  TQM  goals  and  behaviors."  This  goal  discusses  the  need  to 
revise  existing  performance  incentives  to  encourage  and  reward 
continuous  improvement  efforts.  [Ref.  26 :p.  10] 

One  of  the  nine  mid-term  goals  applies  directly  to 
procurement:  "Implementation  commitment  by  major  Defense 
contractors,  with  'critical  mass'  achieved  in  at  least  the  top 
2  5  contractors."  This  goal  intends  to  result  in  a  commitment 
from  the  top  2  5  Defense  contractors  to  independently  implement 
continuous  process  improvement  programs.  These  2  5  contractors 
receive  contracts  valuing  more  than  50  percent  of  DOD's  total 
acquisition  expenditures,  and  they  receive  most  of  DOD  major 
system  contracts.  This  commitment  is  expected  to  involve  the 
training  of  senior  leadership  within  these  firms  in  TQM  and 
their  active  involvement  in  continuous  improvement  efforts. 
[Ref.  16:p.  6] 

In  the  long  term,  one  of  the  four  goals  of  the  Master 
Plan  impacts  upon  procurement:  "Widespread  Defense  industry 
implementation  of  continuous  process  improvement." 
Implementation  of  this  goal  involves  employment  of  TQM  by  all 
DOD  prime  and  subcontractors.  Its  achievement  requires  three 
things  of  DOD.  First,  DOD  must  encourage  and  assist 
contractors  in  their  implementation  of  continuous  process 
improvement.   Second,  DOD  is  expected  to  develop  criteria  to 

44 


evaluate  contractor  commitment  to  continuous  improvement. 
Finally,  the  degree  and  success  of  contractor  process 
improvement  efforts  are  to  be  included  in  source  selection 
strategies.  [Ref.  26:p.  4]  Within  this  goal,  the  Master  Plan 
states,  "The  intent  of  continuous  process  improvement  efforts 
is  to  recognize  and  reward  dedication  and  adherence  to  the 
improvement  process,  and  not  restrict  reward  to  attaining  a 
specific  standard  of  performance."   [Ref.  16:p.  4] 

E .   SUMMARY 

This  chapter  briefly  described  three  DOD  contractor 
quality  programs.  The  programs  described  were  the  Air  Force 
CPARS ,  NAC  Indianapolis  Blue  Ribbon  Contractor  program,  and 
DOD  TQM  initiative.  The  procedures  associated  with  each 
program  have  been  provided  as  background  information  to 
support  the  next  chapter.  The  chapter  that  follows  will 
discuss  the  merits  and  shortcomings  of  the  three  programs  and 
their  relationship  to  RYG. 
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IV.   A  COMPARISON  OF  RED  YELLOW  GREEN  AND  OTHER 
CONTRACTOR  QUALITY  INITIATIVES 


A.  INTRODUCTION 

This  chapter  will  use  the  background  information  provided 
in  Chapter  III  to  analyze  the  Red  Yellow  Green  (RYG)  program 
relative  to  the  other  three  programs  described — the  Contractor 
Performance  Assessment  Reporting  System  (CPARS) ,  the  Blue 
Ribbon  Contractor  program,  and  the  Total  Quality  Management 
(TQM)  initiative.  The  purpose  of  the  analysis  is  to  determine 
whether  RYG  should  co-exist  with  these  other  initiatives  or  if 
it  should  replace  or  be  replaced  by  any  one  of  them. 

The  assessment  will  be  organized  in  the  same  way  as  the 
background  information  presented  in  Chapter  III.  First,  RYG 
will  be  compared  to  CPARS.  Next,  RYG  will  be  evaluated 
relative  to  Naval  Avionics  Center  (NAC)  Indianapolis'  Blue 
Ribbon  Contractor  program.  Finally,  RYG  will  be  assessed  for 
its  compatibility  with  the  Department  of  Defense  (DOD)  TQM 
initiative. 

B.  CONTRACTOR  PERFORMANCE  ASSESSMENT  REPORTING  SYSTEM 

The  CPARS  devised  by  the  Air  Force  is  intended  for  use  in 
source  selections  for  major  systems  contracts.  Therefore,  the 
threshold  for  employment  of  CPARS  is  set  at  $5  million  and  the 
number  of  contractors  included  in  the  program  has  been  limited 
to  84.   The  Air  Force  had  two  major  reasons  for  setting  these 
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limits.  The  first  involves  maintenance  of  the  Contractor 
Performance  Assessment  Reports  (CPARs) .  The  second  involves 
the  nature  of  the  information  reported  in  the  CPARs  and, 
therefore,  the  way  in  which  this  information  must  be  used  in 
the  source  selection  process.  These  two  factors  will  be 
discussed  in  detail  in  the  following  paragraphs. 

Maintenance  of  CPARs  requires  many  resources.  First,  it 
involves  the  time  invested  by  the  program  manager  to  prepare 
the  report.  Although  the  reports  are  supposed  to  be  based  on 
facts,  they  are  not  merely  collections  of  data.  They  are 
written,  subjective  assessments  of  the  contractor's  perfor- 
mance and,  therefore,  require  some  effort  on  the  part  of  the 
program  manager.  The  second  factor  concerning  maintenance  of 
CPARs  involves  the  efforts  of  those  who  verify  the  information 
contained  in  the  reports,  ensure  that  the  reports  are  current, 
and  physically  store  the  reports.  Since  the  CPARs  are 
expected  to  be  used  in  future  source  selections,  the  Air  Force 
carefully  verifies  the  information  contained  in  the  reports  to 
ensure  that  it  is  compatible  with  past  trends  and  other  CPARs 
filed  on  the  same  contractor.  When  an  activity  uses  CPARs  in 
source  selection,  the  currency  of  the  information  is  checked. 
If  a  more  current  CPAR  is  desired,  the  procuring  activity 
contacts  the  program  office  and  requests  more  up-to-date 
information.  In  addition,  since  the  system  is  not  yet 
automated,  maintaining  hard  copy  files  of  all  CPARs  at  the  six 
Product   Divisions   within   the   Air   Force   Systems   Command 
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requires  a  tremendous  clerical  effort.   Maintenance  of  the 
CPARs  is  a  very  labor  intensive  process. 

The  information  contained  in  CPARs  is  subjective.  It  is 
based  on  facts,  but  much  of  it  is  still  the  opinion  of  the 
program  manager  and  members  of  the  program  office. 
Additionally,  because  a  CPAR  is  a  written  report,  it  is  a 
relatively  long  document.  Because  of  its  subjectivity  and 
length,  the  use  of  CPARs  in  source  selection  requires  careful 
evaluation  of  the  information  contained  in  the  report.  To 
perform  this  evaluation,  the  CPARs  must  first  be  collected, 
evaluated  for  their  relevance  to  the  procurement  in  question, 
and  assembled  into  an  integrated  report.  The  resulting  report 
is  then  further  evaluated  in  conjunction  with  other  past 
performance  data  to  determine  a  contractor's  level  of  risk. 
This  is  a  relatively  long  process.  The  Air  Force  believes  the 
process  is  beneficial  for  major  weapon  system  procurements. 
However,  CPARS  is  not  used  for  non-major  acquisitions. 

The  points  discussed  above  suggest  that  CPARS  would  not  be 
appropriate  for  contracting  operations  at  field  activities  for 
three  reasons. 

First,  the  investment  in  resources  to  maintain  the  CPARs 
could  not  be  justified  by  the  relatively  small  dollar 
procurements  awarded  by  field  contracting  activities.  The 
return  on  investment  to  obtain  higher  quality  material  would 
surely  be  less  than  the  cost  to  prepare  and  maintain  the 
CPARs . 
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Second,  in  order  to  be  effective  in  field  contracting,  the 
number  of  contractors  included  in  the  program  would  have  to  be 
greatly  expanded.  The  contractors  that  compete  in  major 
system  procurements  are  relatively  few.  However,  field 
contracting  offices  deal  with  thousands  of  contractors.  Even 
if  fewer  than  100  contractors  were  included  in  the  system, 
field  activities  would  find  it  very  difficult  to  obtain 
accurate  past  performance  information.  An  expansion  of  CPARS 
at  the  field  level  to  accommodate  such  a  large  number  of 
contractors  would  be  an  impossible  undertaking. 

Finally,  even  if  CPARS  were  expanded,  field  contracting 
offices  would  have  neither  the  time  nor  the  resources  to 
perform  the  evaluation  necessary  to  correctly  apply  it  to 
source  selection. 

In  summary,  CPARS  seems  to  be  only  appropriate  for  use  in 
the  procurement  of  major  systems.  It  should  not  be  considered 
for  expanded  use  in  field  contracting.  RYG  was  designed  for 
use  by  field  contracting  offices.  Accordingly,  its  formula 
driven  assessment  of  past  performance  would  not  be  appropriate 
for  use  in  major  system  procurements.  The  two  programs  were 
designed  to  satisfy  different  needs  in  different  areas  of 
Government  contracting.  Therefore,  neither  should  be 
considered  as  a  replacement  for  the  other. 
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C.   BLUE  RIBBON  CONTRACTOR  PROGRAM 

The  Blue  Ribbon  Contractor  program  at  NAC  Indianapolis  has 
five  unique  features.  First,  it  is  an  automated  system 
created  by  NAC  using  data  collected  by  NAC.  Second,  the 
criteria  used  to  define  a  Blue  Ribbon  contractor  and  the 
programs  included  in  the  program  were  chosen  to  focus 
attention  on  the  critical  and  problem  materials  procured  by 
NAC.  Third,  contractors  must  apply  for  Blue  Ribbon  status  to 
be  included  in  the  program,  and  NAC  performs  the  analysis  to 
determine  whether  these  contractors  qualify  for  the  benefits 
of  the  program.  Fourth,  the  program  concentrates  on 
commodities  important  to  NAC ' s  mission,  and  a  contractor  must 
make  a  delivery  to  NAC  at  least  once  each  year  to  be  included 
in  the  program.  Therefore,  the  contractors  designated  as  Blue 
Ribbon  will  generally  be  the  contractors  NAC  deals  with  on  a 
regular  basis.  Fifth,  use  of  the  program  in  source  selection 
requires  judgment.  Unlike  RYG,  it  is  not  a  formula-driven 
program.  The  decision  to  award  to  a  Blue  Ribbon  contractor 
over  a  lower-priced  offeror  (not  included  in  the  program) 
requires  an  assessment  by  both  contracting  and  quality 
assurance  personnel.  Therefore,  the  two  groups  must 
communicate  with  each  other  to  use  the  program  effectively. 

Like  RYG,  the  Blue  Ribbon  Contractor  program  is  ap- 
propriate for  use  in  field  contracting.  However,  a  Blue 
Ribbon  program  requires  that  an  activity  devote  many  more 
resources  than  does  RYG.   First,  the  activity  would  need  the 
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capability  to  automate.  Second,  it  would  have  to  devise  the 
criteria  to  be  used  in  designating  Blue  Ribbon  contractors. 
NAC  chose  delivery,  but  other  commands  might  be  more 
interested  in  quality  or  reliability.  Third,  it  would  have  to 
decide  which  commodities  are  important  enough  to  its  mission 
to  justify  inclusion  in  the  program.  Fourth,  it  would  need  to 
devote  resources  to  the  program  to  evaluate  applicants  and 
maintain  the  Blue  Ribbon  Contractor  list.  Finally,  the 
contracting  and  quality  assurance  personnel  in  the  command 
would  have  to  be  willing  and  able  to  work  together  to 
implement  the  program. 

Some  commands  would  be  unable  to  take  the  five  steps 
discussed  in  the  previous  paragraph.  They  might  lack  the 
automation  necessary  to  implement  the  program.  Or, 
contracting  personnel  might  be  physically  located  away  from 
quality  assurance  personnel.  This  would  be  the  case  for  a 
Navy  Regional  Contracting  Center  procuring  goods  and  services 
for  many  of  its  Navy  field  activities.  In  either  case, 
implementation  of  a  Blue  Ribbon  Contractor  program  would  be 
impractical.  Therefore,  a  Blue  Ribbon  program  does  not  seem 
to  be  a  viable  option  for  every  field  contracting  activity. 

The  question  then  is  whether  RYG  and  Blue  Ribbon 
Contractor  programs  could  co-exist  at  activities  capable  of 
implementing  both  programs.  This  researcher  believes  the 
answer  to  that  question  is  yes — the  programs  are  compatible 
and  can  enhance  each  other.   If  used  together,  the  Blue  Ribbon 
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program  would  be  used  when  procuring  goods  and  services  that 
the  command  considers  critical  to  the  performance  of  its 
mission.  The  Blue  Ribbon  program  would  allow  the  command  to 
concentrate  its  attention  on  those  parameters  that  it 
considers  important.  RYG  could  be  used  for  all  other  procure- 
ments. By  definition,  RYG  procurements  would  be  for  less 
critical  and  less  frequently  purchased  materials.  These  items 
would  not  justify  the  resources  needed  to  apply  the  Blue 
Ribbon  program.  In  addition,  the  command  may  not  have  been 
able  to  generate  enough  local  data  on  critical  items  to  assess 
contractor  past  performance.  RYG  would  allow  the  command  to 
consider  past  performance  in  source  selection  while  employing 
a  very  simple  system. 

Selected  field  activities  could  benefit  from  using  both 
RYG  and  a  tailored  Blue  Ribbon  Contractor  program.  The  most 
likely  evolution  would  be  to  first  implement  RYG  since  it  is 
less  complex.  Use  of  RYG  should  help  personnel  involved  in 
the  source  selection  process  become  comfortable  with  using  an 
automated  system.  In  addition,  implementation  of  RYG  should 
force  contracting  and  quality  assurance  personnel  to  work 
together.  RYG  requires  quality  assurance  personnel  to  use 
past  performance  to  evaluate  offerors  and  determine  the 
quality  assurance  actions  required  of  each.  They  must  then 
provide  this  assessment  to  contracting.  Therefore,  RYG 
involves  quality  assurance  more  deeply  in  the  source 
evaluation  process.     Both  will   help  pave   the   way   for 
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introduction  of  a  Blue  Ribbon  Contractor  program  concentrating 
attention  on  the  area  of  the  command's  specialization.   [Ref. 

27] 

D.   TOTAL  QUALITY  MANAGEMENT 

The  discussion  in  this  section  will  focus  on  whether  RYG 
furthers  the  goals  of  TQM.  The  purpose  of  this  discussion  is 
to  evaluate  whether  RYG  is  consistent  with  DOD ' s  TQM 
initiative  or  whether  RYG  hinders  this  initiative.  This 
section  will  first  consider  RYG ' s  conformance  to  the  general 
goals  of  TQM  as  outlined  in  Chapter  III.  .  Second,  it  will 
relate  RYG  to  the  goals  in  the  DOD  TQM  Master  Plan. 

1.   TOM  and  RYG — General 

Two  of  Dr.  Deming's  14  principles  were  determined  to 
directly  impact  the  Government  procurement  process  and, 
therefore,  apply  to  the  comparison  of  TQM  and  RYG.  These  two 
principles  are: 

1.  Number  3.  "Cease  dependence  on  mass  inspection  to 
achieve  quality.  Eliminate  the  need  for  inspection  on 
a  mass  basis  by  building  quality  into  the  product  in  the 
first  place."   [Ref.  20:p.  23] 

2.  Number  4.  "End  the  practice  of  awarding  business  on  the 
basis  of  price  tag.  Instead,  minimize  total  cost.  Move 
toward  a  single  supplier  for  any  one  item,  on  a  long- 
term  relationship  of  loyalty  and  trust."  [Ref.  20 :p. 
23] 

The  first  concept  addresses  the  requirement  for  contractors  to 

practice  statistical  process  control   (SPC) .    The  second 

addresses  the  relationship  between  buyers  and  suppliers. 
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Implementation  of  it  involves  considering  total  cost  when 
awarding  contracts  by  ensuring  that  suppliers  practice  TQM. 

RYG  does  not  satisfy  all  of  the  requirements  of  these 
two  principles,  nor  does  it  require  contractors  to  practice 
all  of  Deming's  14  principles.  It  does,  however,  help  the 
Government  end  the  practice  of  awarding  business  on  the  basis 
of  price  tag  by  considering  the  total  cost  of  a  procurement 
when  the  cost  of  poor  quality  is  added  to  the  source  selection 
formula.  Therefore,  in  a  limited  way,  RYG  helps  to  further 
the  goals  of  TQM. 

This  researcher  believes  that  a  limited  implementation 
of  TQM  in  field  contracting  is  probably  a  more  realistic 
undertaking  for  DOD.  Even  the  third  and  fourth  Deming 
principles  will  be  very  difficult  to  implement  since  they 
require  two  resources  that  may  not  be  available  to  field 
contracting  officers — time  and  personnel. 

■  ■  Time  is  required  to  ensure  that  the  supplier  practices 
TQM.  Contracting  personnel  would  have  to  examine  the  TQM 
programs  of  offerors  in  detail.  They  would  need  to  ensure 
that  the  contractors  practice  SPC  as  well  as  other  Deming 
principles.  Implementation  of  the  management  principles  is 
more  difficult  and  more  time  consuming  because  it  involves 
evaluating  the  firm's  daily  management  style.  Field 
contracting  requires  the  processing  of  many  procurement 
actions  in  a   relatively  short  period  of  time.   Contracting 
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offices  do  not  devote  the  time  necessary  to  perforin  such  an 
on-going  evaluation. 

In  addition  to  time,  personnel  are  required  to  perform 
a  vendor  TQM  assessment.  As  discussed  in  Chapter  III,  the 
buying  activity  must  be  able  to  evaluate  whether  the  supplier 
is  taking  the  correct  measurements  at  the  critical  points  in 
production  and  if  the  sampling  technique  is  proper.  This 
requires  an  in  depth  knowledge  of  both  the  production  process 
and  statistics.  Someone  with  technical  knowledge  of  the  item 
being  purchased,  perhaps  an  engineer,  would  have  to  be  a 
permanent  member  of  the  evaluation  team.  Additionally,  a 
statistician  would  probably  be  needed.  It  is  unlikely  that  a 
contracting  office  would  have  either  the  number  of  personnel 
or  skill  levels  required  for  the  task. 
2.   TOM  and  RYG — POD 

With  the  exception  of  its  short-term  goals,  the 
segments  of  the  DOD  TQM  Master  Plan  relating  to  procurement 
seem  to  focus  on  procurement  of  major  systems.  This  is 
probably  due  to  the  dollar  values  involved  in  major  system 
procurements  as  well  as  the  degree  of  management  attention 
focused  on  each  program.  The  expected  returns  from  using  TQM 
for  major  system  acquisitions  would  be  greater  than  for 
smaller  procurements.  In  addition,  the  time  schedule  of  a 
major  system  acquisition  would  more  easily  allow  the 
contractor  evaluation  required  by  TQM.  Procurements  made  in 
the  field  do  not  involve  such  high  dollar  values.   Neither  do 
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they  allow  the  degree  of  management  attention  required  by- 
procurement  of  a  major  system.  However,  in  spite  of  the  fact 
that  the  Master  Plan  does  not  specifically  address  contractor 
quality  programs  intended  for  field  contracting,  this 
researcher  believes  that  RYG  can  still  be  shown  to  conform  to 
the  intent  of  the  Master  Plan. 

In  the  short  term,  the  Master  Plan  discusses  enlisting 
defense  industry  commitment  to  TQM  by  using  programs  already 
in  existence  within  DOD.  Additionally,  it  hopes  to  employ 
recognition  and  reward  systems  based  on  TQM  goals  and 
behaviors.  [Ref.  26 :p.  10]  RYG  is  consistent  with  both  these 
objectives. 

First,  RYG  employs  an  existing  system,  the  Product 
Deficiency  Reporting  and  Evaluation  Program  (PDREP) ,  to 
improve  the  quality  of  material  received  by  the  Navy.  It 
expands  the  application  of  this  system  to  allow  greater 
consideration  of  contractor  past  performance  in  the  source 
selection  process.  Second,  RYG  could  be  used  to  reward 
contractors  who  provide  good  quality  material  to  the 
Government.  It  accomplishes  this  by  considering  the  cost  of 
poor  quality  when  making  source  selection  decisions.  Since 
proposals  from  contractors  with  good  quality  history  are  not 
assessed  a  Technical  Evaluation  Adjustment,  these  contractors 
have  a  competitive  advantage  over  those  with  a  poor  quality 
history. 
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The  Master  Plan's  procurement  goal  for  the  mid  term 
addresses  only  major  system  procurements.  It  involves  getting 
a  commitment  from  DOD's  top  2  5  contractors  to  implement  TQM. 
These  contractors  receive  more  than  50  percent  of  DOD's 
procurement  dollars  and  most  of  its  major  system 
contracts.  [Ref.  26:p.  6]  This  goal  is  not  related  to  RYG 
since  RYG  neither  helps  nor  hinders  its  implementation. 

In  the  long  term,  the  Master  Plan  seeks  employment  of 
TQM  by  all  DOD  prime  and  subcontractors  [Ref.  26:p.  4].  As 
discussed  earlier  in  this  section,  it  is  difficult  for  field 
contracting  offices  to  require  and  verify  employment  of  TQM  by 
the  vast  number  of  contractors  dealt  with  at  this  level. 
However,  RYG  at  least  allows  them  to  consider  improved 
quality,  which  is  the  result  of  continuous  process 
improvement,  in  their  source  selection  decisions.  Therefore, 
use  of  RYG  is  at  least  a  step  toward  TQM  implementation  in 
field  contracting.  Additionally,  implementing  a  Blue  Ribbon 
Contractor  program  in  conjunction  with  RYG  can  move  field 
contracting  activities  even  closer  to  the  goals  of  TQM. 

As  described  in  the  Blue  Ribbon  Contractor  Program 
section  of  this  chapter,  the  RYG  and  Blue  Ribbon  Contractor 
programs  can  enhance  each  other.  RYG  is  necessary  for 
procurement  of  infrequently  used  and  non-critical  materials 
that  do  not  justify  a  great  deal  of  management  attention.  It 
is  also  needed  by  commands  incapable  of  implementing  a  Blue 
Ribbon  Contractor  program.    RYG  focuses  on  the  negative 
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aspects  of  poor  performance  to  improve  quality.  A  Blue  Ribbon 
Contractor  program  requires  more  resources  to  implement  than 
does  RYG  and  so  would  be  used  in  the  procurement  of  materials 
an  activity  considers  critical.  Blue  Ribbon  programs  focus  on 
the  positive  aspects  of  past  performance  to  improve  quality. 
Because  of  its  simplicity  and  focus  on  the  negative, 
RYG  could  be  considered  a  first  step  toward  implementing  TQM 
in  field  contracting.  The  program  will  always  be  needed  when 
a  Blue  Ribbon  program  is  either  unavailable  or  inappropriate. 
Since  the  Blue  Ribbon  Contractor  programs  are  more  complex  and 
focus  on  the  positive,  they  are  a  logical  second  step  toward 
implementing  TQM  in  field  contracting. 

E .   SUMMARY 

This  chapter  evaluated  the  three  DOD  quality  programs 
described  in  the  previous  chapter — CPARS ,  the  Blue  Ribbon 
Contractor  program,  and  the  TQM  initiative.  It  compared  and 
contrasted  these  programs  to  RYG  and  discussed  whether  RYG 
should  co-exist  with  them.  The  next  chapter  will  assess  the 
strengths  and  weaknesses  of  the  RYG  program.  The  analysis 
will  include  a  discussion  concerning  the  program's  ability  to 
withstand  a  protest.  It  will  also  assess  the  current  status 
of  the  RYG  test. 
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V.   STRENGTHS  AND  WEAKNESSES  OF  RED  YELLOW  GREEN 

A.   INTRODUCTION 

This  chapter  will  attempt  to  assess  the  strengths  and 
weaknesses  of  the  Red  Yellow  Green  (RYG)  program.  First,  the 
four  areas  of  the  program  most  open  to  contractor  protests 
will  be  addressed.  The  first  area  considered  will  be  whether 
RYG  results  in  de  facto  debarment  of  Red  and  Yellow  contrac- 
tors. The  second  consideration  is  whether  the  Technical 
Evaluation  Adjustments  (TEAs)  could  be  regarded  as  penalties 
and  whether  it  is  fair  for  the  Government  to  consider  costs  of 
quality  (a  basic  premise  of  RYG)  in  the  source  selection 
decision  in  the  first  place.  The  third  to  be  discussed  will 
be  whether  consideration  of  past  performance,  as  an  evaluation 
factor,  constitutes  a  responsibility  determination.  The 
fourth  area  addressed  in  this  chapter  will  be  the  accuracy  of 
the  data  base  used  to  rate  contractors  Red,  Yellow,  or  Green. 
The  overall  discussion  will  include  an  evaluation  of  the 
program's  ability  to  withstand  a  contractor  protest. 

Next,  the  progress  of  the  RYG  program  test  will  be 
examined.  This  discussion  will  first  involve  an  assessment  of 
test  implementation  at  each  of  the  five  test  activities.  It 
will  then  provide  data  documenting  the  results  of  the  test  to 
date  and  discuss  the  significance  of  these  data. 
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B.   PROTEST  ISSUES 

1 .   De  Facto  Debarment 

The  discussion  concerning  de  facto  debarment  will 

center  around  whether  designating  a  contractor  as  Red  or 

Yellow  has  the  same  effect  as  debarment.    The  effect  of 

debarment  as  described  in  the  Federal  Acquisition  Regulation 

(FAR)  is: 

(a)  Debarred  or  suspended  contractors  are  excluded  from 
receiving  contracts,  and  agencies  shall  not  solicit  offers 
from,  award  contracts  to,  or  consent  to  subcontracts  with 
these  contractors,  unless  the  acquiring  agency's  head  or  a 
designee  determines  that  there  is  a  compelling  reason  for 
such  action....   [Ref.  28] 

Cibinic  and  Nash  provide  a  similar  description  of  the  effect 

of  debarment: 

The  second  broad  sanction  is  debarment  of  the  contractor 
or  its  employees  from  taking  any  Government  contracts  for  a 
stated  period  of  time. ...  Debarment  precludes  contracts  with 
any  agency  of  the  federal  government  for  a  period  of  up  to 
three  years....   [Ref.  29] 

Based  on  the  above  definition,  it  is  clear  that  debarment  of 

contractors  excludes  them  from  competing  for  any  Government 

contract.    This  means  that  the  Government  must  avoid  both 

soliciting  offers  from  and  awarding  contracts  to  debarred 

contractors. 

Application  of  the  RYG  test  procedures  neither  stops 

the  Government  from  soliciting  offers  from  nor  awarding 

contracts  to  Red  and  Yellow  contractors  [Ref.  30].   It  merely 

allows  the  Government  to  consider  the  costs  it  expects  to 

incur  based  on  the  quality  of  material  it  is  most  likely  to 


60 


receive  from  an  offeror.  If,  after  considering  the  costs 
related  to  additional  quality  assurance  requirements,  a  Red  or 
Yellow  contractor  is  the  lowest-priced  responsible  offeror, 
the  Red  or  Yellow  contractor  will  be  awarded  the  contract. 

Since  the  Government  will  be  soliciting  proposals 
from,  and  awarding  contracts  to.  Red  and  Yellow  contractors, 
application  of  the  RYG  test  procedures  is  not  considered  a  de 
facto  debarment  in  the  analysis  of  the  researcher.  It  is 
expected  that  the  RYG  program  will  be  able  to  withstand  any 
protest  in  this  area. 
2.   Use  of  TEAS 

The  second  area  for  potential  protest  is  the  use  of 
TEAS.  The  propriety  of  using  TEAs  will  be  considered  first 
by  discussing  whether  TEAs  could  be  considered  penalties  and, 
therefore,  unenforceable,  and  second  by  discussing  whether  it 
is  fair  for  the  Government  to  consider  costs  of  quality  when 
evaluating  proposals.  In  order  to  assess  the  reasonableness 
of  the  values  assigned  as  TEAs,  the  researcher  considered 
precedent  associated  with  the  reasonableness  of  liquidated 
damages  amounts. 

Regulations  and  precedent  applying  to  liquidated 
damages  were  used  to  determine  whether  the  TEAs  (as  calculated 
in  the  RYG  test  procedures)  could  be  considered  penalties. 
The  FAR  states  that  "liquidated  damages  fixed  without  any 
reference  to  probable  actual  damages  may  be  held  to  be  a 
penalty,  and  therefore  unenforceable. "   [Ref.  28:para.  12.202] 
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Based  on  this  regulation,  the  question  then  becomes  whether 

the  TEAS  are  related  to  actual  costs  that  the  Government  would 

expect  to  incur  when  forced  to  carry  out  additional  quality 

assurance  requirements. 

Cibinic  and  Nash  state,  "The  Government  contract  rule 

is  that  the  reasonableness  of  the  forecast  of  liquidated 

damages  is  evaluated  by  looking  at  the  situation  at  the  time 

the  contract  is  made...."    [Ref.  29: p.  802]   They  further 

explain  that  even  liquidated  damages  calculated  at  a  fixed 

rate  of  dollars  per  day  have  been  upheld  by  the  courts  because 

they  are  considered  to  cover  the  Government's  administrative 

expenses,  "i.e.,  inspection,  superintendence  or  engineering 

costs,   which   the   Government  will   incur   if   a   delay   is 

encountered."     [Ref.   29:  p.   803]    In  the  case  of  Young 

Associates,  Inc.  v.  United  States,  200  Ct.  CI.  438,471  F.2d 

618  (1973)  the  court  stated: 

The  answer  is,  we  think,  that  the  regulation  does  not 
require  a  liquidated-damage  schedule  to  be  tailor  made  for 
each  individual  contract.  It  is  enough  if  the  amount 
stipulated  is  reasonable  for  the  particular  agreement  at  the 
time  it  is  made.   [Ref.  29:p.  804] 

In   summary,   amounts   of   liquidated   damages   are 

considered  reasonable  if  they  are  a  fair  estimate  (calculated 

when  the  contract  is  made)  of  the  administrative  costs  the 

Government  would  expect  to  incur  during  a  delay.   Further,  the 

estimate  need  not  be  tailored  to  each  contract.    Based  on 

this,  amounts  of  TEAs  should  be  considered  reasonable  if  they 

are  truly  a  fair  estimate  of  the  administrative  costs  the 
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Government  expects  to  incur  because  a  contractor  has  poor 
quality  history. 

The  TEAS  furnished  as  examples  in  Chapter  II  and 
Appendix  D  were  calculated  using  Navy  Material  Quality 
Assessment  Office  (NMQAO)  historical  data.  These  data  include 
labor  hours  required  for  the  quality  assurance  actions,  labor 
rates,  and  costs  of  testing.  [Ref.  2]  They  are,  however, 
provided  only  as  examples.  Each  test  activity  has  used  these 
examples  and  their  own  quality  assurance  history  and  labor 
rates  to  calculate  local  TEAs.  Therefore,  the  TEAs  assessed 
in  the  test  of  RYG  are  tailored  to  the  expected  costs  of  each 
activity.  Since  the  TEAs  represent  a  fair  estimate  of  the 
costs  each  activity  expects  to  incur  because  of  increased 
quality  assurance  requirements  (calculated  at  the  time  of 
contract  award) ,  this  researcher  believes  that  they  would  be 
considered  reasonable  in  the  event  of  a  protest.  In  addition, 
since  the  TEAs  assessed  correspond  to  the  specific  quality 
assurance  actions  required  in  the  contract,  they  could  be  con- 
sidered to  be  tailored  to  each  contract.  Based  on  the  case 
law  concerning  liquidated  damages,  a  tailored  assessment  might 
not  be  necessary,  however,  it  probably  strengthens  the 
argument  justifying  the  reasonableness  of  assigned  TEAs. 

The  second  consideration  in  the  assessment  of  the 
propriety  of  using  TEAs  is  whether  it  is  fair  for  the 
Government  to  consider  the  costs  of  additional  quality 
assurance   requirements   in   source   selection.     There   is 
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precedent  for  considering  costs  similar  to  TEAs  in  the  source 
selection  decision.  Specifically,  the  FAR  discusses  the 
consideration  of  first  article  test  and  transportation  costs 
in  the  source  selection  process. 

Regarding  first  article  testing,  the  FAR  states, 
"Solicitations  containing  a  testing  and  approval  requirement 
shall ...  include,  when  the  Government  is  responsible  for  first 
article  testing,  the  Government's  estimated  testing  costs  as 
a  factor  for  use  in  evaluating  offers...."  [Ref.  28: para. 
9.306]  In  addition,  the  FAR  requires  that  the  solicitations 
"inform  offerors  that  the  requirement  may  be  waived  when 
supplies  identical  or  similar  to  those  called  for  have 
previously  been  delivered  by  the  offeror  and  accepted  by  the 
Government...."  [Ref.  28: para.  9.306]  These  two  statements 
suggest  that  not  only  should  the  Government  consider  the  costs 
of  testing  in  the  source  selection  decision,  but  that  the 
decision  concerning  the  requirement  of  first  article  testing 
should  be  based  on  the  performance  history  of  the  offerors. 
Both  support  the  use  of  TEAs  when  evaluating  past  performance. 

Regarding  transportation  costs,  the  FAR  states, 
"Solicitations,  when  appropriate,  shall  specify  that  offers 
may  be  f.o.b.  origin,  f.o.b.  destination,  or  both;  and  that 
they  will  be  evaluated  on  the  basis  of  the  lowest  overall  cost 
to  the  Government."  [Ref.  28:para.  47.305-2]  The  FAR  further 
states,  "When  evaluating  offers,  contracting  officers  shall 
consider     transportation    and    transportation-related 
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costs..."  [Ref.  28:para.  47.306]  These  statements  strengthen 
the  argument  that  it  is  fair  for  the  Government  to  use  TEAs 
based  on  differences  in  the  costs  it  will  incur  because  of 
quality  differences. 

3 .   Use  of  Responsibility-Related  Factors 

The  argument  against  use  of  past  performance  as  a 
factor  in  source  evaluation  states  that  since  past  performance 
is  one  of  seven  responsibility  factors  listed  in  the  FAR,  its 
use  in  source  evaluation  constitutes  a  responsibility 
determination  [Ref.  28:para.  9.104-1].  In  the  case  of  small 
businesses,  a  determination  of  non-responsibility  must  be 
referred  to  the  cognizant  Small  Business  Administration  (SBA) 
Regional  Office  [Ref.  28:para.  19.602-1].  Therefore,  if  use 
of  past  performance  in  source  evaluation  were  considered  to 
constitute  responsibility  determination,  any  negative  impact 
the  RYG  test  procedures  had  on  small  business  would  have  to  be 
referred  to  the  SBA.  Failure  to  comply  with  FAR  Part  19.602- 
1  would  violate  statute  and  regulation.  In  order  to  determine 
whether  the  responsibility  argument  is  valid,  several 
pertinent  General  Accounting  Office  (GAO)  protest  decisions 
have  been  examined.  The  cases  and  their  implications  are 
summarized  below. 

Three  of  the  cases  evaluated  indicate  that 
responsibility  factors  may  be  used  in  source  evaluation 
provided  that  negotiated  procurement  practices  are  used,  and 
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the   selecting   activity   uses   these   factors   to   make   a 

comparative  evaluation  of  offerors. 

In  the  matter  of  B  &  W  Service  Industries,  Inc.  (B  & 

W)  ,  B  &  W  contended  that  responsibility  factors  (i.e.,  company 

experience,   past  performance,   and  the  experience  of  key 

personnel)  should  not  have  been  used  in  the  Department  of 

Housing  and  Urban  Development's  evaluation  of  their  proposal. 

GAO  denied  the  protest.    [Ref.  31]    In  its  decision,  GAO 

stated: 

Contracting  officers  evaluate  prospective  contractors  to 
determine  their  responsibility,  that  is,  their  capability  to 
perform  the  work.... COG  (Certificate  of  Competency) 
referrals  to  SBA  are  only  required  where  contracting 
officers  find  small  businesses  to  be  nonresponsible.  In 
this  case,  the  agency  did  not  find  the  protester 
nonresponsible,  but  considered  its  proposal  to  be  weak  under 
some  evaluation  factors  listed  in  the  solicitation. 

With  regard  to  these  factors,  it  is  not  improper  in  a 
negotiated  procurement  to  include  traditional  responsibility 
factors  among  the  technical  evaluation  criteria. .. .As  long 
as  the  factors  are  limited  to  areas  which,  when  evaluated 
comparatively,  can  provide  an  appropriate  basis  for  a 
selection  that  will  be  in  the  government's  best  interest, 
COG  procedures  do  not  apply  to  a  technical  proposal 
deficient  in  those  areas.   [Ref.  31] 

In  the  case  of  Utah  Geophysical  Inc.   (UGI)  ,  UGI 

contended  that  the  Nuclear  Regulatory  Commission  (NRG)  failed 

to  refer  a  non-responsibility  determination  to  the  SBA.   They 

argued  that  because  source  evaluation  included  responsibility 

factors   (i.e.,   experience   and   level   of   effort   for  key 

personnel  assigned  to  the  project),  the  NRG ' s  finding  that 

their  proposal  was  unacceptable  constituted  a  determination  of 
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non-responsibility.    [Ref.  32]   GAO  denied  the  protest  and 

stated  in  its  decision: 

Matters  which  are  normally  considered  in  responsibility 
determinations  may  properly  be  considered  in  evaluation  of 
proposals  when  negotiation  procedures  are  used  and  agency 
makes  relative  assessment  of  competing  offerors'  ability  to 
meet  contract  requirements.   [Ref.  32] 

In  the  matter  of  Electrospace  Systems,  Inc.  (ESI),  ESI 

protested  the  Army's  use  of  offerors'  experience  in  source 

evaluation.     This   evaluation   criterion   impacted   ESI's 

exclusion  from  the  competitive  range.   They  argued  that  use  of 

responsibility-related  factors  in  source  evaluation  violated 

the  SBA's  statutory  authority  to  certify  the  responsibility  of 

small  businesses.    [Ref.  33]   In  its  decision,  GAO  cites  a 

previous  case  involving  SBD  Computer  Services  Corporation,  B- 

186950,   December   21,   1976,   76-2   CPD   511.     In   its   1976 

decision,  GAO  stated: 

In  many  other  cases,  we  have  recognized  that  contracting 
agencies  may  properly  utilize  evaluation  factors  which 
include  experience  and  other  areas  that  would  otherwise  be 
encompassed  by  offeror  responsibility  determinations  when 
the  needs  of  those  agencies  warrant  a  comparative  evaluation 
of  those  areas.   [Ref.  33] 

In  its  decision  to  deny  ESI's  protest,  GAO  stated: 

Since  neither  10  U.S.C.  2304(g)  nor  applicable  regulation  in 
any  way  restrict  the  "other  factors"  that  may  be  used  by 
agencies  in  selecting  the  proposal  having  the  greatest  value 
to  the  Government,  we  have  not  prohibited  procuring  agencies 
from  using  responsibility-related  factors  in  making  relative 
assessments  of  the  merits  of  competing  proposals.   [Ref.  33] 

The   cases   cited   above   all   support   use   of 

responsibility  factors  in  source  evaluation.   However,  care 

must  be  exercised  to  ensure  that,  if  such  factors  are  used. 


67 


the  Government  makes  a  comparative  evaluation  of  offerors,  not 

a  pass/fail  determination.   If  a  responsibility  factor  is  used 

to  ascertain  the  technical  acceptability  of  a  proposal,  its 

use   will   be   considered   to   constitute   a   responsibility 

determination.     Two   cases   providing   examples   of   this 

distinction  are  described  below. 

In  the  case  of  Sanford  and  Sons  Company  (Sanford)  , 

Sanford  protested  the  Army  Corps  of  Engineers*  determination 

that  their  proposal  was  technically  unacceptable.  The  request 

for  proposals  (RFP)  stated  that  the  contract  would  be  awarded 

to  the  lowest-priced,  technically-qualified  offeror.   Two  of 

the  six  criteria  for  source  evaluation  detailed  in  the  RFP 

were:   (1)    organization   and   administration;    and    (2) 

satisfactory  performance  record.   The  Army  found  Sanford 's 

proposal  to  be  technically  unacceptable  in  both  organization 

and  administration  and  past  performance.   Both  of  these  areas 

are  considered  responsibility  factors.    GAO  sustained  the 

protest.   [Ref.  34]   In  its  decision  GAO  stated: 

We  have  cautioned  that  an  agency  may  not  find  that  a  small 
business  is  nonresponsible  under  the  guise  of  a  relative 
assessment  of  responsibility  factors  and  thus  avoid 
referring  the  matter  to  the  Small  Business  Administration. 
[Ref.  34] 

The  decision  further  states: 

Although  an  agency  may  use  traditional  responsibility 
factors,  like  prior  performance,  as  technical  evaluation 
factors  where  its  needs  warrant  a  comparative  evaluation  of 
proposals,  an  agency's  rejection  of  a  small  business  firm's 
offer  as  unacceptable  under  such  factors  was  improper  where 
the  agency's  decision  did  not  reflect  a  relative  assessment 
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of  the  offer  but  instead  effectively  constituted  a  finding 
of  nonresponsibility .   [Ref.  34] 

In   the   matter   of   Siems   International   Electron 

Microscope  Service  (Siems) ,  Siems  protested  the  cancellation 

of  a  RFP  by  the  National  Institutes  of  Health  (NIH) .   NIH  had 

determined  that  Siems  was  non-responsible  under  the  RFP,  but 

Siems  was  subsequently  issued  a  Certificate  of  Competency  by 

the  SBA.    NIH  stated  that  the  solicitation  was  cancelled 

because  the  specification  in  the  RFP  was  inadequate,  not 

whether  or  not  Siems  was  responsible.    NIH  determined, 

therefore,  that  the  requirement  should  be  resolicited.   Under 

the  new  RFP,  Siems  proposal  was  determined  to  be  technically 

unacceptable.      This   determination   was   based   on   a 

responsibility  factor — contractor  capacity.    [Ref.  35]   GAO 

sustained  the  protest,  stating: 

Where  offerors  were  not  required  to  submit  technical 
proposals  to  service  electron  microscopes  but  only  to  offer 
to  conform  to  the  best  practices  of  the  industry,  and  the 
factors  making  up  the  technical  criteria  were  evaluation  of 
capacity  factors,  the  determination  an  offeror  was 
technically  unacceptable  amounted,  in  essence,  to  a 
determination  of  nonresponsibility  for  reasons  of  capacity 
that  required  a  referral  to  the  Small  Business 
Administration. . . .   [Ref.  35] 

The  conclusion  that  can  be  drawn  after  examination  of 

the  above  five  cases  is  that  responsibility  factors  may  be 

used  in  source  evaluation,  but  that  activities  must  be  careful 

not   to   use   such   factors   to   determine   the   technical 

acceptability  of  a  proposal.   If  implemented  as  written,  the 

RYG  test  procedures  do  not  use  past  performance  to  determine 
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technical  acceptability.  It  will  be  used,  however,  to  better 
determine  the  true  cost  to  the  Government  of  doing  business 
with  each  offeror.  In  fact,  the  test  procedures  specifically 
state  that  a  contractor's  status  under  the  test  is  not  to  be 
used  as  a  substitute  for  the  requirement  to  make  a 
responsibility  determination  [Ref.  l:encl.  (1),  p.  4].  The 
test  activities,  however,  must  exercise  care  in  administering 
the  RYG  procedures  to  ensure  that  they  do  not  use  a 
contractor's  past  performance  to  judge  his  technical 
acceptability. 

In  summary,  the  Government  faces  a  risk  that  GAO  may 
sustain  a  protest  arguing  that  use  of  the  RYG  test  procedures 
constitutes  a  responsibility  determination.  However,  if  the 
test  procedures  are  implemented  as  intended,  the  Office  of 
General  Council  (OGC)  in  the  Office  of  the  Assistant  Secretary 
of  the  Navy  (Shipbuilding  and  Logistics)  (ASN(S&L))  believes 
that  the  Government  has  a  good  argument  to  support  its  case 
that  use  of  past  performance  under  the  RYG  test  constitutes  a 
comparative  evaluation  of  offerors,  not  a  responsibility 
determination  [Ref.  30]. 

4 .   Accuracy  of  the  Data  Base 

The  validity  of  the  Contractor  Evaluation  System  (CES) 
data  base  presents  the  greatest  potential  for  protest  since 
they  are  the  basis  of  categorizing  contractors  as  Red,  Yellow, 
or  Green.  The  data  are  maintained  very  well  by  the  Naval 
Material  Quality  Assessment  Office  (NMQAO) .   Therefore,  the 
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CES  is  an  accurate  portrayal  of  the  data  received  by  NMQAO 
from  the  sources  described  in  Chapter  II.  [Ref.  36]  Problems 
will  occur,  however,  if  the  data  are  inaccurate  when  they  are 
submitted  to  NMQAO.  Culpability  (Government  or  contractor) 
for  rejects  and  deficiencies  is  assigned  before  the  data  are 
submitted  to  NMQAO.  Because  of  the  volume  of  data  received, 
it  would  be  impossible  for  NMQAO  to  verify  each  item  [Ref. 
36].  If  an  activity  submits  inaccurate  data,  a  contractor 
could  be  mistakenly  classified  Red  or  Yellow.  If  a  contractor 
were  to  lose  a  contract  because  of  an  incorrect 
classification,  he  would  have  a  strong  argument  to  support 
filing  a  protest. 

Two  issues  arise  in  evaluating  the  probability  that 
the  RYG  test  procedures  will  survive  a  protest  based  on 
inaccurate  data.  First,  the  current  accuracy  of  the  data  base 
must  be  considered.  Prior  to  NMQAO  classifying  contractors 
Red,  Yellow,  or  Green,  letters  were  mailed  to  all  Red  and 
Yellow  contractors  informing  them  of  their  status  and  the 
reasons  for  the  classification.  With  each  monthly  update  of 
the  data  base,  contractors  are  notified  of  any  change  in  their 
status  under  the  test.  Contractors  were  advised  to  contact 
NMQAO  if  they  had  any  questions  about  the  program  or  their 
color  classification.  NMQAO  mailed  2037  notification  letters 
and  has  received  247  oral  and  written  responses  to  the 
classifications  assigned.  Thirteen  of  these  responses 
resulted  in  changes  to  the  data  base.   In  three  cases,  data 
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corrections  resulted  in  a  change  of  color  classification. 
[Ref.  37]  Thirteen  corrections  out  of  2037  would  suggest  a 
0.56  percent  error  rate.  However,  such  an  assumption  would  be 
invalid  since  only  a  small  percentage  of  the  contractors 
notified  provided  a  response. 

The  only  way  to  estimate  the  accuracy  of  the  data  base 
is  to  draw  a  representative  sample  of  contractors  and  verify 
the  data  leading  to  their  color  classification.  Such  an 
undertaking  would  require  more  resources  than  are  currently 
available  to  NMQAO.  Additionally,  even  if  such  an  evaluation 
were  to  be  made,  there  is  no  yardstick  available  to  determine 
what  would  be  an  acceptable  level  of  accuracy.  [Ref.  36] 
Therefore,  the  validity  and  acceptable  accuracy  level  of  the 
data  base  can  only  be  determined  during  the  test. 

One  of  the  benefits  of  gaining  approval  from  the 
Defense  Acquisition  Regulatory  Council  for  the  one-year  test 
of  the  RYG  program  is  that  GAO  has  traditionally  declined  to 
hear  protests  on  other  test  programs  [Ref.  38].  It  is  hoped 
that  a  test  period  free  of  protests  will  allow  NMQAO  to  assess 
the  validity  of  the  data  and  resolve  inaccurate  data,  and 
hopefully  expand  the  program  to  all  of  DOD. 

The  second  issue  pertaining  to  the  accuracy  of  the 
data  base  concerns  the  notification  letters  sent  to 
contractors  informing  them  of  their  status  in  the  program. 
These  letters  inform  contractors  of  their  current  color 
classification  based  on  documented  performance  history.   In 
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the  opinion  of  Ms.  Sandra  Desbrow-Jensen  of  the  OGC  at 
ASN(S&L)  ,  if  a  contractor  does  not  respond  within  a  reasonable 
period  to  correct  errors  in  the  data  base,  he  will  be  estopped 
from  later  protesting  the  validity  of  the  data  base  [Ref.  30]. 
If  GAO  supports  this  position,  the  contractor  notification 
letters  should  help  eliminate  protests  based  on  the  accuracy 
of  the  data. 

C.   PROGRESS  OF  RYG  TEST 
1.   Implementation 

In  order  to  assess  the  progress  of  the  RYG  program 
implementation,  the  researcher  conducted  interviews  with 
personnel  at  each  of  the  five  test  activities.  The  purpose  of 
the  interviews  was  to  gain  the  perspective  of  both  contracting 
and  quality  assurance  personnel  involved  in  RYG 
implementation.  Therefore,  interviews  were  conducted  with 
individuals  from  both  areas  at  each  activity. 

The  following  paragraphs  summarize  the  comments 
received  during  the  interviews.  The  perspective  of 
contracting  personnel  are  presented  first,  followed  by 
comments  from  the  quality  assurance  community.  Finally,  the 
researcher  discusses  similarities  and  differences  between 
these  comments  and  their  implications. 

a.   Contracting 

(1)   Naval  Supply  Center  (NSC)  Pensacola.    The 
researcher  interviewed  CDR  Walsh,  Contracting  Officer  at  NSC 
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Pensacola  when  the  RYG  implementation  began.  He  explained 
that  the  NSC  had  begun  RYG  implementation  with  a  small  number 
of  Federal  Supply  Classes  (FSCs) .  The  Contracting  Department 
chose  2  0  FSCs  from  a  list  given  to  them  by  the  Naval  Aviation 
Depot  (NADEP)  Pensacola.  Initially,  they  have  applied  the  RYG 
test  procedures  to  ten  of  the  20  FSCs  and  small  purchases. 
They  intend  to  later  expand  the  program  to  all  2  0  FSCs  and 
major  procurements.  The  test  has  not  generated  a  great  deal 
of  activity  to  date,  but  CDR  Walsh  anticipates  a  greater 
impact  when  the  program  is  expanded.   [Ref.  39] 

The  test  procedures  have  not  yet  resulted  in 
significant  extra  work.  Quality  assurance  personnel  at  the 
NADEP  have  helped  absorb  some  of  the  extra  burden.  The  NSC 
has  not  received  much  negative  feedback  from  contractors. 
[Ref.  39] 

CDR  Walsh  expressed  concern  about  the 
accuracy  of  the  data  base  used  for  the  RYG  test. 
Additionally,  he  worries  about  the  affect  the  program  will 
have  on  distributors  who  are  also  small  businesses.  Although 
theoretically  distributors  should  be  responsible  for  the 
quality  of  material  they  sell,  small  businesses  may  not  have 
the  capability  to  properly  test  their  stock.  He  fears  that 
the  RYG  program  may  hurt  these  small  businesses.   [Ref.  39] 

(2)  NSC  Charleston.  Mrs.  Davidson,  Deputy 
Director,  Purchase  Division,  Regional  Contracting  Department, 
NSC  Charleston,  was  interviewed  to  gain  the  perspective  of 
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contracting  personnel  at  the  other  NSC  involved  in  the  RYG 
test.  She  explained  that  the  RYG  test  procedures  are  being 
applied  to  procurements  of  Controlled  Industrial  Material 
(CIM)  for  the  Charleston  Naval  Shipyard.  All  solicitations 
for  CIM  are  done  in  writing,  including  small  purchases,  so  it 
takes  30  days  to  receive  proposals.  As  a  result,  no  awards 
have  yet  been  made  using  the  RYG  test  procedures.   [Ref.  40] 

Mrs.  Davidson  believes  that  evaluation  of 
proposals  using  the  test  procedures  will  involve  extra  work 
for  contracting  personnel.  However,  since  no  evaluations  have 
yet  been  performed,  she  does  not  know  how  burdensome  this 
extra  requirement  will  be.  She  thinks  the  program  will  be 
beneficial  because  it  will  allow  the  NSC  to  resolve  quality 
problems  before  contract  award;  therefore,  she  expects  RYG  to 
reduce  contract  administration  activities.   [Ref.  40] 

Mrs.  Davidson  expressed  concern  that 
application  of  the  RYG  procedures  would  have  a  negative  impact 
on  the  department's  Productive  Unit  Resourcing  System  (PURS) 
statistics  because  it  adds  work  before  contract  award  which 
will  not  be  given  appropriate  credit.  Time  savings  are  not 
realized  until  the  contract  administration  phase  of  the 
procurement  process.  Therefore,  although  she  anticipates 
benefits  from  the  program,  she  wonders  whether  they  will 
outweigh  the  program's  negative  impacts.   [Ref.  40] 

(3)   Ships  Parts  Control   Center   (SPCC) .    The 
researcher  interviewed  Mr.  Minahan,  Director  of  the  Hull, 
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Mechanical,  and  Electrical  Contracting  Department,  SPCC.  He 
explained  that  SPCC  has  applied  the  RYG  test  procedures  to 
five  FSCs.  So  far,  his  department  has  not  been  burdened  with 
much  extra  work.   [Ref.  41] 

Mr.  Minahan  indicated  that  the  procurements 
included  in  the  RYG  test  at  SPCC  involve  fairly  high  dollar 
values.  As  a  result,  he  does  not  believe  the  application  of 
TEAS  will  have  much  impact  on  their  source  selection 
decisions.  He  thinks  the  TEAs  may  have  a  greater  effect  at 
other  activities  making  smaller  dollar  value  procurements. 
[Ref.  41] 

(4)  Naval  Air  and  Enqineerincr  Center  (NAEC)  .  Mr. 
Fackenthal,  Supervisory  Contract  Specialist  at  NAEC  Lakehurst, 
was  interviewed.  He  explained  that  NAEC  is  applying  the  RYG 
small  purchase  method  as  outlined  in  the  test  procedures. 
However,  they  requested  and  received  approval  to  follow  the 
Fixed  Price-Greatest  Value  procedures  instead  of  the 
procedures  using  TEAs  for  their  major  procurements.  NAEC  made 
this  request  after  reviewing  a  sample  of  the  contractors  they 
have  done  business  with  during  the  last  two  years.  Their 
review  indicated  that  only  ten  percent  of  the  contractors  in 
the  sample  were  in  the  CES  data  base.  NAEC  attributes  this  to 
the  nature  of  their  procurements.  They  make  very  few 
repetitive  buys.  Many  of  their  procurements  have  production 
lead  times  of  up  to  one  year.  Therefore,  the  performance 
history  generated  is  too  old  to  classify  a  contractor  as  Red, 
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Yellow,  or  Green.  NAEC  believes  that  the  more  subjective 
Fixed  Price-Greatest  Value  procedures  will  better  suit  their 
procurements.   [Ref.  42] 

Mr.  Fackenthal  indicated  that  it  is  difficult 
to  assess  the  results  of  the  program  yet  because  they  only 
began  using  the  small  purchase  procedures  in  October  1989  and 
have  not  yet  made  a  contract  award  using  the  major  purchase 
procedures.  However,  he  stated  that  all  the  comments  he  has 
received  about  the  program,  including  those  from  contractors, 
have  been  positive.  He  indicated  that  the  contracting 
personnel  at  NAEC  are  positive  about  the  RYG  program.  They 
like  the  idea  of  being  able  to  use  past  performance  in  source 
evaluation.  He  foresees  that  the  program  will  encourage 
contractors  with  poor  performance  records  to  improve  the 
quality  of  the  material  they  deliver  to  the  Government.  [Ref. 
42] 

(5)  Naval  Avionics  Center  (NAC) .  The  researcher 
interviewed  Mr.  Wilson,  Manager  of  Acquisition  and  Improvement 
at  NAC.  He  commented  that  the  RYG  test  was  progressing  very 
well  at  NAC,  and  that  they  are  considering  expanding  the 
number  of  FSCs  procured  under  the  program.  They  are  trying  to 
tailor  the  test  procedures  to  their  operation.   [Ref.  43] 

Mr.  Wilson  indicated  that  the  RYG  test  has 
not  resulted  in  a  great  deal  of  extra  work  for  contracting 
personnel  at  NAC.  He  believes  that  the  ease  with  which  they 
implemented  the  program  is  due  in  part  to  the  culture  at  NAC 
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which  welcomes  innovation.  He  would  like  to  see  the  CES 
expanded  to  the  Buyer  Information  System  described  in  Chapter 
II.  He  hopes  that  the  validity  of  the  data  base  can  be  proven 
during  the  test  so  that  in  the  future  it  can  be  used  by 
contracting  personnel  to  perform  responsibility 
determinations.   [Ref.  43] 

Mr.  Wilson  explained  that  most  of  NAC's 
contractor  quality  problems  have  been  eliminated  by 
implementation  of  the  Blue  Ribbon  Contractor  program. 
Therefore,  they  have  not  been  able  to  determine  whether  the 
RYG  test  program  has  caused  contractors  to  improve  the  quality 
of  the  material  they  supply  to  NAC.  He  also  indicated  that 
field  activities  can  make  errors  impacting  on  a  contractor's 
color  classification.  In  one  instance  at  NAC,  some  test 
equipment  was  improperly  programmed.  The  error  resulted  in 
the  erroneous  rejection  of  material  supplied  by  a  vendor  on 
NAC's  Blue  Ribbon  Contractor  list.  The  rejects  at  NAC  caused 
the  contractor  to  be  classified  Red  in  the  RYG  data  base.  The 
contractor  did  not  lose  any  contracts  because  of  his  incorrect 
color  classification.  However,  it  took  NAC  a  couple  of  months 
to  investigate  and  correct  the  problem  after  the  contractor 
questioned  his  color  classification.  [Ref.  43] 
b.   Quality  Assurance 

(1)  NADEP  Pensacola.  Mr.  Hargett,  Quality 
Developmental  Project  manager  at  NADEP  Pensacola,  was 
interviewed  to  gain  perspective  on  the  progress  of  the  RYG 
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test  implementation  in  Pensacola.  Mr.  Hargett  explained  that 
Pensacola  had  purposely  limited  the  initial  application  of  the 
RYG  test  procedures  to  allow  personnel  involved  to  become 
accustomed  to  them.  Their  slow  start  has  resulted  in  little 
activity  from  the  test  to  date.  However,  involvement  in  the 
test  program  has  caused  them  to  increase  the  amount  of  quality 
data  submitted  to  NMQAO.  As  a  result,  they  have  added 
approximately  1000  vendors  to  the  CES  data  base.  He  expects 
this  increase  to  cause  more  activity  under  the  test  in  the 
future.   [Ref.  44] 

Mr.  Hargett  indicated  that  the  RYG  test 
procedures  increase  the  workload  of  the  quality  assurance 
personnel,  but  they  do  not  mind  devoting  the  required  time. 
Because  of  the  test,  Mr.  Hargett  has  become  deeply  involved 
with  the  contracting  personnel  at  NSC  Pensacola.  He  said  this 
is  the  first  time  the  two  organizations  have  worked  together. 
Mr.  Hargett  has  been  involved  with  the  RYG  program  since  its 
conception.  Although  he  was  positive  about  the  test  at  the 
outset,  he  stated  that  he  is  even  more  positive  about  the 
program  now.   [Ref.  44] 

(2)  Charleston  Naval  Shipyard.  The  researcher 
interviewed  Ms.  Cassell,  Supervisory  Quality  Assurance 
Engineer  at  the  Charleston  Naval  Shipyard.  She  indicated  that 
the  test  is  beginning  slowly  there.  She  believes  that  most  of 
the  delay  has  been  caused  by  unrealistically  low  cost 
estimates  placed  on  purchase  requests  by  the  Shipyard's 
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material  planners.  She  explained  that  often  the  planners 
estimate  costs  to  be  less  than  $2500.  As  a  result,  the 
purchase  requests  are  not  included  in  the  RYG  test.  When 
material  is  procured,  however,  the  actual  costs  are  usually 
greater  than  $2500.  Therefore,  procurements  that  should 
qualify  for  the  RYG  test  are  not  included  because  of 
inaccurate  cost  estimates.  She  indicated  that  the  Quality 
Assurance  Department  at  the  Shipyard  has  been  working  with  the 
Planning  Department  to  correct  this  problem.   [Ref.  45] 

Ms.  Cassell  is  positive  about  the  RYG 
program.  She  believes  it  highlights  the  Shipyard's  quality 
assurance  problems  to  the  contracting  personnel  at  NSC 
Charleston.  She  also  thinks,  however,  that  the  location  of 
contracting  and  quality  assurance  in  different  commands  causes 
the  Contracting  Department  at  the  NSC  to  be  out  of  touch  with 
the  quality  problems  experienced  by  the  Shipyard.  She 
believes  this  separation  may  hinder  the  implementation  of  the 
RYG  test.   [Ref.  45] 

(3)  SPCC.  The  researcher  interviewed  Mr.  Jornov, 
Quality  Assurance  Specialist,  Level  1/SS  Section,  SPCC.  He 
indicated  that  SPCC  has  used  the  RYG  procedures  in 
approximately  30  solicitations,  but  has  not  yet  received  any 
material  under  the  program.  So  far,  they  have  not  received 
any  complaints  from  contractors.  He  explained  that  they  have 
limited  the  implementation  to  cover  only  spare  parts.   They 
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later  expect  to  expand  the  program  to  include  valves,  their 
major  commodity.   [Ref.  46] 

Mr.  Jornov  said  that  the  increased  work  load 
has  not  been  burdensome  for  them,  involving  approximately  one 
hour  each  day.  However,  the  work  load  will  increase  if  the 
program  is  expanded.  He  thinks  the  RYG  program  is  worthwhile. 
He  believes  it  needs  some  fine  tuning,  but  that  eventually  it 
will  encourage  vendors  to  improve  the  quality  of  their 
material.  He  indicated  that  he  has  misgivings  about  the  data 
base.  He  is  concerned  that  some  activities  may  not  send  data 
to  NMQAO  and  that  NMQAO  may  not  enter  all  the  data  received. 
He  fears  that  an  incomplete  data  base  may  distort  contractor 
color  classifications.   [Ref.  46] 

(4)  NAEC.  Mr.  Robert  Armitage,  Quality  Assurance 
Specialist,  Procurement  Division,  Quality  Assurance 
Department,  NAEC,  was  interviewed.  He  indicated  that  the 
extra  work  required  by  the  RYG  test  has  not  been  too 
burdensome  for  quality  assurance  personnel  at  NAEC.  He  is 
very  positive  about  the  program.  He  believes  it  is  a  useful, 
viable  tool  for  NAEC.  He  would  like  to  see  the  impact  the 
program  has  on  contractors  with  poor  quality  history 
strengthened.  However,  he  is  glad  that  the  program  allows 
NAEC  to  consider  quality  history  to  a  greater  degree  than  they 
had  in  the  past.  He  said  that  any  new  tool  that  helps  them  do 
their  job  better  is  welcome.   [Ref.  47] 


(5)  NAC.  The  researcher  interviewed  Ms.  Powell, 
Electronics  Engineer,  Quality  Assurance  Department,  NAC.  She 
indicated  that  the  bulk  of  the  extra  work  created  by  the  RYG 
test  has  been  related  to  handling  contractor  complaints. 
Investigating  complaints  generally  takes  between  one  and  two 
weeks.  Ms.  Powell  believes  the  program  is  an  excellent 
concept.  It  forces  commands  to  correct  their  data.  In 
addition,  before  the  RYG  test,  contractors  did  not  know  (or 
care)  whether  their  quality  history  was  good.  Now  they  do. 
Implementation  forces  Government  activities  to  communicate 
with  contractors.   [Ref.  48] 

Ms.  Powell  indicated  that  some  of  the 
contractor  complaints  have  proven  to  be  valid.  Therefore,  she 
believes  the  weakest  part  of  the  program  is  the  data  base. 
Specifically,  she  thinks  problems  may  occur  in  placing 
liability  (Government  or  contractor)  for  quality  deficiencies. 
[Ref.  48] 

c.   Analysis  of  Interviews 

None  of  the  people  interviewed  exhibited  a 
negative  attitude  concerning  the  RYG  test  program.  In 
general,  quality  assurance  personnel  appeared  to  be  more 
positive  about  the  program  than  contracting  personnel. 
Although  the  test  requires  extra  work  of  both  communities, 
quality  assurance  personnel  seemed  more  convinced  that  the 
benefits  derived  from  the  program  would  outweigh  the  extra 
effort.   Perhaps  their  perspective  results  from  a  feeling  that 
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for  the  first  time  they  have  a  tool  to  influence  the  quality 
of  material  before  it  is  received.  The  amount  of  inspection 
and  documentation  quality  assurance  performs  varies  inversely 
with  the  quality  of  material  received.  Therefore,  they 
probably  perceive  that  improved  quality  will  simplify  receipt 
inspection  and  reduce  documentation  of  quality  deficiencies. 
This  will  make  their  job  easier. 

Two  of  the  people  involved  in  contracting, 
however,  were  as  positive  about  the  program  as  were  quality 
assurance  personnel.  These  two  people  work  at  the  two  Navy 
Industrial  Fund  activities  involved  in  the  test,  NAC 
Indianapolis  and  NAEC  Lakehurst.  The  two  contracting  people 
from  the  NSCs  at  Pensacola  and  Charleston  were  the  two  most 
concerned  about  the  program.  One  of  the  reasons  for  the 
difference  in  opinion  on  this  program  was  based  on  the  PURS 
used  at  NSCs.  As  explained  by  Mrs.  Davidson  of  NSC 
Charleston,  this  system  rewards  the  NSC  for  the  speed  with 
which  they  award  contracts  and  purchase  orders.  The  RYG  test 
procedures  slow  down  the  award  process,  promising  improved 
quality  at  the  end  of  the  procurement  process.  Therefore, 
implementation  of  the  RYG  program  may  penalize  activities 
subject  to  PURS  guidelines.  A  second  reason  for  the 
difference  in  attitudes  may  be  the  location  of  the  contracting 
and  quality  assurance  personnel  at  each  test  site. 
Contracting  personnel  at  NSC  Pensacola  must  work  with  quality 
assurance  personnel  at  NADEP  Pensacola;  contracting  personnel 
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at  NSC  Charleston  must  work  with  quality  assurance  personnel 
at  the  Charleston  Naval  Shipyard.  The  NSCs  and  industrial 
activities  they  support  have  different  missions  and  different 
commanding  officers.  These  distinctions  probably  make  it  more 
difficult  for  the  two  functional  areas  to  understand  the 
other's  concerns  and  to  make  the  RYG  program  succeed. 

Aside  from  helping  the  activities  receive  better 
quality  material,  the  interviewees  foresee  two  peripheral 
consequences  of  the  RYG  program  that  are  positive.  First, 
they  expect  the  program  to  encourage  Red  and  Yellow 
contractors  to  improve  the  quality  of  material  they  supply  to 
the  Government.  Second,  they  indicated  that  implementation  of 
the  test  procedures  has  forced  contracting  and  quality 
assurance  personnel  to  work  together.  The  increased  contact 
is  expected  to  help  the  two  groups  better  understand  each 
other's  problems.  The  end  result  should  be  that  both  groups 
are  able  to  do  their  jobs  more  effectively. 

The  biggest  area  of  concern  voiced  by  the  test 
activities  is  the  validity  of  the  data  base.  This  worry  was 
expressed  by  NSC  Pensacola,  both  quality  assurance  and 
contracting  at  NAC  Indianapolis,  and  quality  assurance  at 
SPCC.  This  concern  corresponds  to  the  assessment  of  personnel 
at  the  Naval  Supply  Systems  Command  and  OGC  at  ASN(S&L) 
(described  earlier  in  this  chapter)  that  the  greatest 
potential  for  protest  of  the  program  lies  with  the  accuracy  of 
the  data  base. 
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2  .   Test  Results 

The  one-year  test  of  the  RYG  program  began  1  August 
1989.  Not  all  of  the  five  activities  began  using  the  RYG  test 
procedures  on  this  date,  however.  Moreover,  when  they  began 
the  test,  several  of  the  activities  limited  the  initial  number 
of  procurements  included  in  the  test.  Limited  implementation 
allows  the  activities  to  fine  tune  their  internal  procedures 
before  attempting  full-scale  implementation.  Although  this 
approach  is  based  on  sound  judgment,  it  has  the  peripheral 
effect  of  limiting  the  amount  of  available  data  documenting 
the  results  of  the  RYG  test. 

The  researcher  has  collected  the  data  that  are 
currently  available;  they  are  summarized  below.  Although  the 
data  are  scant,  an  attempt  has  been  made  to  analyze  them  and 
assess  their  implications  on  the  progress  of  the  RYG  test. 
The  data  pertaining  to  the  number  of  Red  (R)  ,  Yellow  (Y)  , 
Green  (G)  ,  and  Insufficient  Data  (I)  contractors  will  be 
presented  first.  Next,  data  concerning  the  status  of 
procurements  under  the  program  will  be  reported. 

Table  1  summarizes  the  numbers  of  letters  sent  to 
contractors  advising  them  of  their  initial  color 
classification  and  subsequent  status  changes  and  the  responses 
received  by  NMQAO.  The  data  in  Table  1  indicate  that  the 
initial  color  classification  resulted  in  1033  Red  and  Yellow 
contractors.   Subsequent  monthly  updates  to  the  RYG  data  base 
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TABLE  1 
RED  YELLOW  GREEN  NOTIFICATION  AND  RESPONSE 


Month 

Noti 

f ication 

Sent 

Response  Received 

May 

1033 

179 

July 

653 

23 

August 

348 

35 

September 

273 

10 

Total 

2307 

247 

Source: 

[Ref . 

37:p. 

9; 

changed  contractor  color  classifications.   Each  update  has 
resulted  in  fewer  contractors  changing  status. 

Table  2  describes  the  categories  into  which  the 
responses  fell  and  the  action  taken  by  NMQAO  in  reaction  to 
them.  The  data  in  Table  2  indicate  that  contractors  disagreed 
with  the  data  or  their  color  classification  in  49  cases  (40 
disputed  the  data  and  nine  disagreed  with  their  color 
classification) .  NMQAO  determined  that  their  disagreements 
were  valid  and  changed  the  data  in  13  cases.  Three  of  the  13 
data  base  corrections  resulted  in  a  change  in  color 
classification.  Although  2307  contractors  were  notified,  only 
247  (or  10.7  percent)  responded.  Therefore,  the  errors 
discovered  because  of  these  responses  cannot  be  used  to 
estimate  the  accuracy  of  the  data  base. 
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TABLE  2 

RESPONSES — CATEGORIES  AND  ACTION  TAKEN 

Category  of  Response  Quantity 

Information  141 

Unaware  of  Rejects  52 

Disputed  Data  4  0 

Classification  Disagreement  9 

Action  Taken 

No  Action 

Forwarded  Data  Profiles 

Referred  to  Other  Command 

Conducted/Scheduled  Meeting 

Changed  Data 

Changed  Classification 

Source:   [Ref.  37:p.  9] 

The  first  group  of  procurements  made  under  the  RYG 
test  are  summarized  in  NMQAO ' s  September  1989  status  report. 
The  data  relating  to  these  procurements  are  detailed  in  Table 
3.  The  data  in  Table  3  indicate  that  in  most  cases  the 
lowest-priced  proposals  were  submitted  by  Green  and 
Insufficient  Data  contractors.  Of  the  77  procurements  made 
under  the  RYG  test,  Red  and  Yellow  contractors  submitted  the 
lowest-priced  proposals  in  only  ten  instances  (seven  displace- 
ments of  Red  and  Yellow  contractors  plus  three  awards  to 
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Ouanti 

■ty 

83 

108 

48 

5 

13 

3 

TABLE  3 
RED/YELLOW/GREEN  PROCUREMENTS 


Awards  to  RYGI     Contractor 
Contractors      Displacements 


Activity 

R 

Y 

G 

I 

R 

Y 

NAEC  Lakehurst 

0 

0 

0 

0 

0 

0 

NAC  Indianapolis 

0 

3 

27 

17 

0 

5 

SPCC  Mechanicsburg 

0 

0 

18 

0 

2 

0 

NSC  Charleston 

0 

0 

0 

0 

0 

0 

NSC  Pensacola 

0 

0 

0 

12 

0 

0 

Total 

0 

3 

45 

29 

2 

5 

Source:   [Ref.  37:pp.  1-6] 

Yellow  contractors) .  These  ten  cases  represent  13  percent  of 
the  77  procurements.  The  awards  made  to  Yellow  contractors 
indicate  that  the  Government's  consideration  of  the  costs  of 
quality  does  not  preclude  contractors  with  poor  quality- 
histories  from  receiving  Government  contracts.  In  other 
words,  a  contractor's  color  classification  does  not  eliminate 
him  from  the  competitive  range. 

It  is  too  early  to  determine  whether  the  RYG  program 
will  improve  either  the  quality  of  material  the  Government 
receives  or  contractors'  motivation  to  deliver  high  quality 
material.  It  is  also  impossible  to  judge  the  reaction 
contractors  will  have  to  displacements  caused  by  the  program. 
The  data  do  show,  however,  that  the  program  is  having  some 
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affect  on  contract  awards;  Red  and  Yellow  contractors  have 
lost  contracts  to  higher  priced  contractors  with  better 
performance  histories.  The  data  also  show  that  the  test 
activities  are  not  using  the  RYG  data  base  to  eliminate 
contractors  from  the  competitive  range.  This  should  support 
the  Government's  position  that  application  of  the  RYG  formula 
in  source  evaluations  does  not  constitute  a  responsibility 
determination . 

D .   SUMMARY 

This  chapter  assessed  the  strengths  and  weaknesses  of  the 
RYG  program.  To  accomplish  this,  it  first  discussed  the 
ability  of  the  RYG  program  to  withstand  protests  based  on  four 
arguments — de  facto  debarment,  TEAs  as  penalties, 
responsibility  determination,  and  validity  of  the  data  base. 
Second,  it  discussed  the  progress  of  RYG  test  implementation 
at  the  five  test  activities.  Interviews  were  conducted  with 
personnel  involved  with  the  test  at  each  of  the  activities  and 
data  related  to  the  test  were  analyzed.  Interviewees 
generally  felt  that  the  RYG  program  would  yield  positive 
results.  However,  several  of  them  expressed  concern  about  the 
validity  of  the  data  in  the  CES .  The  final  chapter  will 
present  the  conclusions  and  recommendations  of  the  thesis. 
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VI.   CONCLUSIONS  AND  RECOMMENDATIONS 

A.  INTRODUCTION 

This  chapter  will  discuss  the  conclusions  and  recommenda- 
tions resulting  from  the  thesis  research.  After  presenting 
the  conclusions  and  recommendations,  the  research  questions 
will  be  answered.  Finally,  recommendations  will  be  made 
concerning  areas  for  further  research. 

B.  CONCLUSIONS 

1.  The  Red  Yellow  Green  CRYG)  Program  is  Appropriate 
for  Use  in  Field  Contracting  Only 

The  RYG  test  procedures  incorporate  a  formula  driven 

assessment   of   contractor   past   performance   in   source 

evaluation.    The  formula  used  in  the  test  was  designed  to 

assess  contractor  quality  history  for  groups  of  relatively 

simple  commodities.   The  test  procedures  were  intended  to  be 

easy  to  apply  and,  therefore,  involve  very  little  subjective 

input  from  contracting  and  quality-assurance  personnel.  Major 

system  contracting  would  require  a  much  more  subjective 

assessment  of  contractor-quality  history  than  allowed  by  the 

RYG  program. 

2 .  The  Air  Force  Contractor  Performance  Assessment 
Reporting  System  (CPARS)  is  Appropriate  for  Major 
Systems  Procurement  Only 

CPARS   should   not   be   expanded   for   use   in   field 

contracting.   Maintenance  of  the  data  for  CPARS  is  very  labor 


90 


intensive.  The  personnel  resources  required  to  maintain  an 
expanded  CPARS  would  prohibit  including  enough  contractors  in 
the  system  to  make  it  useful  for  field  contracting.  In 
addition,  use  of  CPARS  in  source  evaluation  requires  time  and 
personnel  not  available  in  field  contracting  offices. 

3 .  Blue  Ribbon  Contractor  Programs  are  Appropriate  for 
Use  at  Field  Activities  in  Which  Contracting  and 
Quality  Assurance  Personnel  are  Able  to  Work  Together 

Unlike  RYG,  the  assessment  of  contractor  performance 

history  under  a  Blue  Ribbon  program  is  not  formula-driven. 

The  decision  to  award  to  a  Blue  Ribbon  contractor  over  a 

lower-priced  offeror  (not  included  in  the  program)  requires 

an  assessment  by  both  contracting  and   quality-assurance 

personnel.   Therefore,  the  two  groups  must  communicate  with 

each  other  to  use  the  program  effectively. 

4 .  RYG  and  Blue  Ribbon  Contractor  Programs  Can  Be 
Implemented  Together  and  Can  Enhance  Each  Other 

A  Blue  Ribbon  program  would  be  used  when  procuring 

critical  goods  and  services  because  it  allows  an  activity  to 

concentrate   its   attention   on   those   parameters   that   it 

considers   important.     RYG  would  be  used   for  all   other 

procurements  which  do  not  justify  devotion  of  the  resources 

needed  to  apply  a  Blue  Ribbon  program. 

5 .  The  RYG  Program  (Especially  if  Implemented  in  Conjunc- 
tion with  a  Blue  Ribbon  Contractor  Program)  is  a 
Realistic  Method  for  Initiating  Total  Quality 
Management  (TQM)  in  Field  Contracting 

The  time  and  personnel  required  to  perform  vendor  TQM- 

assessments   make   complete   TQM   implementation   in   field 
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contracting  impractical.  However,  the  RYG  and  Blue  Ribbon 
Contractor  programs  allow  field  contracting  activities  to 
consider  contractor  past  performance  in  the  source  selection 
decision.  Therefore,  they  allow  activities  to  comply,  at 
least  partially,  with  one  of  Deming's  14  principles:  "End  the 
practice  of  awarding  business  on  the  basis  of  price  tag. 
Instead,  minimize  total  cost.  Move  toward  a  single  supplier 
for  any  one  item,  on  a  long-term  relationship  of  loyalty  and 
trust."   [Ref.  20:p.  23] 

6 .  Application  of  the  RYG  Test  Procedures  Does  Not 
Constitute  De  Facto  Debarment 

Use   of   the   test   procedures   neither   stops   the 

Government  from  soliciting  offers  from  nor  awarding  contracts 

to  Red  and  Yellow  contractors.    If,  after  considering  the 

costs  related  to  additional  quality  assurance  requirements,  a 

Red  or  Yellow  contractor  is  the  lowest  priced  responsible 

offeror,  the  Red  or  Yellow  contractor  will  be  awarded  the 

contract. 

7 .  The  Assessment  of  Technical  Evaluation  Adjustments 
(TEAS)  Under  the  RYG  Test  is  Reasonable  and  Does  Not 
Represent  a  Penalty  to  Red  and  Yellow  Contractors 

There  is  precedent  for  considering  costs  similar  to 
TEAS  in  source  evaluation  (i.e.,  consideration  of  first- 
article  test  and  transportation  costs) .  Therefore,  it  is 
reasonable  for  the  Government  to  consider  the  costs  of 
additional  quality  assurance  requirements  in  source  selection. 
TEAS  are  calculated  using  historical  data  for  labor  hours 
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required  for  quality  assurance  actions,  labor  rates,  and  costs 

of  testing.   Since  their  calculation  is  based  on  actual  costs 

the  Government  would  expect  to  incur  when  forced  to  carry  out 

additional  quality  assurance  actions,  TEAs  should  not  be 

considered  penalties. 

8 .   If  the  RYG  Test  Procedures  are  Implemented  as 

Intended,  the  Government  Should  Be  Able  to  Support 
the  Argument  that  the  Use  of  Past  Performance  Under 
the  Test  Does  Not  Constitute  a  Responsibility 
Determination 

Precedent  from  cases  in  which  contractors  protested 

the  use  of  responsibility-related  factors  in  source  evaluation 

indicate  that  responsibility  factors  may  be  used  in  source 

evaluation.   However,  activities  must  be  careful  to  use  such 

factors  to  perform  a  comparative  evaluation  of  offerors,  not 

to  determine  the  technical  acceptability  of  proposals.   The 

RYG  test  procedures  use  past  performance  to  better  determine 

the  true  cost  to  the  Government  of  doing  business  with  each 

offeror,   not  to  determine  the  technical  acceptability  of 

proposals. 

9  .  The  Validity  of  the  Contractor  Evaluation  System  (CES) 
Data  Base  Presents  the  Greatest  Potential  for  Protest 
of  the  RYG  Test  Procedures 

Personnel  at  the  Naval  Supply  Systems  Command  and  at 

several  of  the  activities  implementing  the  RYG  test  expressed 

concern  regarding  the  accuracy  of  the  data  in  the  CES . 

Specifically,  they  fear  that  if  an  activity  were  to  submit 

inaccurate  data,  a  contractor  could  be  mistakenly  classified 

Red  or  Yellow.    If  a  contractor  were  to  lose  a  contract 
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because  of  an  incorrect  classification,  he  would  have  a  strong 

argument  to  support  filing  a  protest. 

10 .   It  is  Possible  that  Red  and  Yellow  Contractors  Will  Be 
Estopped  from  Protesting  the  Validity  of  the  CES 

Each  Red  and  Yellow  contractor  is  notified  of  his 

color  classification  and  the  performance  history  behind  the 

designation.   He  is  given  an  opportunity  to  challenge  the  data 

supporting  his  status  under  the  RYG  program.   If  a  contractor 

is  aware  that  the  data  justifying  his  classification  are 

incorrect,  and  he  does  not  respond  to  correct  them  within  a 

reasonable  period,  he  may  give  up  the  right  to  challenge  these 

errors  later  during  a  protest. 

C.   RECOMMENDATIONS 

1.   Evaluation  of  RYG  at  the  End  of  the  Test  Period 

The  benefits  of  the  RYG  program  and  its  ability  to 
withstand  a  protest  should  be  evaluated  at  the  end  of  the 
program  test,  currently  scheduled  for  1  August  1990.  There 
are  not  enough  data  currently  available  to  evaluate  whether 
the  benefits  of  the  RYG  program  outweigh  the  costs  of 
implementation.  The  positive  comments  expressed  by  personnel 
at  the  five  activities  involved  in  the  test  suggest  that  the 
program  will  be  beneficial.  However,  an  assessment  concerning 
whether  the  program  improves  contractor  performance  must 
eventually  be  made.  Such  an  evaluation  should  include 
reviewing  quality  records  at  the  five  test  activities  to 
determine  whether  the   quality  of   incoming  material   has 
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improved.  In  addition,  a  sample  of  contractors  should  be 
evaluated  to  determine  whether  the  RYG  program  has  encouraged 
them  to  improve  their  performance. 

2 .  Continuation  of  the  RYG  Program  After  Completion  of 
the  Test 

If  there  are  not  enough  data  to  adequately  assess  the 

value  of  the  RYG  program  at  the  end  of  the  test  period,  use  of 

the  test  procedures  should  be  continued  until  the  program  can 

be  properly  evaluated.   Contractor  evaluation,  as  described  in 

the  previous  paragraph,  will  require  the  passage  of  enough 

time  for  contractors  to  become  aware  of  the  effects  of  the  RYG 

program,  change  their  performance,  and  deliver  enough  material 

to  the  Government  to  document  that  change  in  performance.   The 

criteria  used  to  designate  contractors  as  Red,  Yellow,  and 

Green  span  periods  up  to  two  years.   Therefore,  it  is  probable 

that  an  accurate  evaluation  of  the  affect  the  RYG  program  has 

on  contractors  will  require  more  than  the  one  year  of  data 

that  will  be  accumulated  under  the  test.   Since  the  program 

has  the  potential  to  yield  such  positive  results,  its  use 

should  be  continued  until  the  program  can  be  evaluated 

properly. 

3 .  Implementation  of  Blue  Ribbon  Contractor  Programs  with 
RYG 

If  the  RYG  program  is  determined  to  be  beneficial, 

activities  should  be  encouraged  to  implement  a  tailored  Blue 

Ribbon  Contractor  program   in  conjunction  with  RYG.     As 

discussed  in  the  Conclusions  section  of  this  chapter,  RYG  and 
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Blue  Ribbon  Contractor  programs  can  enhance  each  other.  In 
addition,  they  both  move  field  contracting  offices  toward 
implementation  of  TQM.  Therefore,  implementing  them  together 
should  prove  beneficial  to  the  Department  of  Defense  (DOD) . 

D.   ANSWERS  TO  THE  RESEARCH  QUESTIONS 

1.   How  is  the  Navy's  Red  Yellow  Green  Program  Structured 
and  How  is  it  Intended  to  Improve  the  Quality  of 
Material  Procured  by  the  Navy? 

The  RYG  program  uses  contractor  performance  history  in 
source  evaluation.  The  program  uses  this  history  in  one  of 
two  ways.  The  first  approach  uses  a  contractor's  quality 
history  to  determine  whether  the  Government  will  require 
additional  quality  assurance  actions  when  doing  business  with 
the  contractor.  If  the  contractor's  performance  history  is 
good,  additional  quality  assurance  actions  will  not  be 
required;  if  his  history  is  poor,  additional  actions  will  be 
required.  If  additional  quality  assurance  is  required,  the 
costs  of  performance  to  the  Government  are  added  to  the 
contractor's  proposal  using  TEAs.  The  source  selection 
decision  is  made  using  the  proposed  prices  of  Green 
contractors  and  the  adjusted  prices  of  Red  and  Yellow 
contractors. 

The  second  approach  involves  a  more  subjective  use  of 
contractor  performance  history.  A  source  selection  plan  is 
prepared  by  quality  assurance  personnel,  specifying  the 
criteria  to  be  used  in  evaluating  Red,  Yellow,  and  Green 
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contractors.  When  proposals  are  received,  quality  assurance 
personnel  assign  ratings  to  each  offeror  based  on  the  criteria 
outlined  in  the  source  selection  plan.  Contracting  personnel 
award  the  contract  based  on  the  scores  received  by 
contractors,  considering  their  quality  and  price  ratings. 

The  RYG  program  is  intended  to  improve  the  quality  of 
material  procured  by  the  Navy  in  two  ways.  First, 
consideration  of  contractor  quality  history  in  source 
evaluation  should  result  in  a  higher  percentage  of  contract 
awards  to  contractors  with  good  performance  history.  This  is 
expected  to  improve  the  quality  of  material  received.  Second, 
the  program  should  encourage  contractors  with  poor  quality 
histories  to  improve  their  performance  in  order  to  receive 
more  Government  contracts  in  the  future. 

2 .  How  are  Red,  Yellow,  and  Green  Contractors  Defined? 
A  Red  contractor  is  considered  a  high  quality  risk. 
His  performance  history  is  poor.  It  warrants  requirement  of 
special  quality  assurance  actions  and  higher-level  review 
before  contract  award.  A  Yellow  contractor  is  considered  a 
moderate  quality  risk.  His  performance  history  is  poor  enough 
to  warrant  requirement  of  special  quality  assurance  actions. 
A  Green  contractor  is  considered  a  low  quality  risk.  His 
performance  history  is  good  and,  therefore,  warrants  no 
special  quality  assurance  actions. 
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3 .  What  are  the  Other  Current  Contractor  Quality 
Initiatives  within  POD? 

The  other  current  quality  initiatives  within  DOD  are 

the   Air   Force   CPARS,   the   Naval   Avionics   Center   (NAC) 

Indianapolis  Blue  Ribbon  Contractor  program,  and  the  DOD  TQM 

initiative. 

4 .  How  Does  RYG  Compare  and  Contrast  to  These  Other 
Programs? 

RYG  and  CPARS  were  designed  to  be  used  in  different 

areas  of  procurement.   RYG  is  intended  to  be  employed  by  field 

contracting  activities.   CPARS  is  intended  for  major  systems' 

procurement.   RYG  and  NAC ' s  Blue  Ribbon  Contractor  program  are 

both  intended  for  use  in  field  contracting.   The  Blue  Ribbon 

program  requires  an  activity  to  devote  more  resources  to  and 

employ  more  judgment  in  contractor  performance  evaluation  than 

does  RYG.    It  would  be  impractical  for  some  activities  to 

implement   a   Blue   Ribbon   Contractor   program.     However, 

implementation  of  such  a  program  can  enhance  the  results 

expected  of  the  RYG  program.   The  RYG  program  is  consistent 

with  the  goals  of  the  DOD  TQM  initiative.    It  helps  field 

contracting  offices  to  begin  implementation  of  TQM. 

5 .  What  are  the  Strengths  and  Weaknesses  Associated  with 
RYG? 

The  RYG  program  has  many  strengths.   It  allows  use  of 

contractor   performance   history   in   the   source   selection 

decision  and  so  should  improve  the  quality  of  material 

received  by  the  Government.    It  requires  few  resources  to 
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implement  and,  therefore,  can  be  used  by  field  contracting 
offices.  It  forces  contracting  and  quality-assurance 
personnel  to  work  together.  It  forces  Government  activities 
to  communicate  with  industry.  The  program  does  have  some 
weaknesses,  however.  The  data  base  used  in  the  program  is 
very  large,  and  its  accuracy  cannot  be  currently  assessed.  If 
the  data  base  proves  inaccurate,  the  program  is  built  on  a 
weak  foundation.  The  program's  evaluation  of  contractor 
performance  history  is  formula-driven.  Therefore,  it  is  only 
appropriate  for  procurement  of  relatively  simple  commodities. 
6 .   How  is  the  RYG  Program  Being  Tested  in  the  Market? 

The  RYG  program  is  being  tested  at  five  Navy  field 
activities.  Interviews  with  personnel  at  the  test  activities 
indicate  that  the  test  began  slowly  but  is  progressing  well. 
Both  contracting  and  quality  assurance  personnel  expressed 
positive  opinions  about  the  program.  There  are  currently  too 
few  data,  however,  to  assess  the  success  of  the  test. 

E.   AREAS  FOR  FURTHER  RESEARCH 

One  area  for  further  research  is  to  evaluate  the  benefits 
of  the  RYG  program  after  the  one-year  test  period.  More  data 
will  be  available  at  this  point  and,  therefore,  a  researcher 
will  be  better  able  to  assess  the  merits  of  the  program. 
Another  area  for  further  research  is  to  determine  how  NAC '  s 
Blue  Ribbon  Contractor  program  could  be  exported  to  other 
activities.     The   Blue   Ribbon   program   has   proven   very 
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beneficial  for  NAC  and  could  be  expected  to  benefit  other 
field  contracting  activities. 


100 


COLOR 

CODE 

RED 

A 

B 

C 

appendix  a 

red  yellow  green  classification  criteria 

classification  criteria 

on  current  navy  vdar 

method  c,  d,  and/or  e  currently  in  effect 

quality  info  on  latest  pre-award  survey 
(pas)  within  last  year — no  award 

d       latest  product-oriented  survey  (pos)  in 
last  two  years  unacceptable 

e       latest  quality  system  review  (qsr)  in  last 
two  years  unacceptable 

f       latest  special  survey  in  last  two  years 
unacceptable 

g       reject  rate  15%  or  more  in  last  two  years 
for  five  or  more  lots 

h       two  or  more  unsat  first  article  tests 
(fat)  in  last  year 

j       two  or  more  category  "i"  qdrs  in  last  year 

k       six  or  more  category  "ii"  action  qdrs  in 
last  year 

l       six  or  more  contractor  liable  rods  in  last 
year" 

m       six  or  more  contractor  liable  waivers/ 
deviations  in  last  year" 

n       on  dla  contractor  alert  list  for  major 
deficiencies" 


Classification  criteria  L,  M,  N  for  "Red"  and  "Yellow" 
have  not  yet  been  incorporated. 
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COLOR    CODE  CLASSIFICATION  CRITERIA 

YELLOW    A  ISSUED  VDAR  LETTER  OF  CONCERN 

B  PREVIOUSLY  CLASSIFIED  "RED" 

C  LATEST  PAS  IN  LAST  TWO  YEARS — AWARD  WITH 

FINDINGS 

D  LATEST  POS  IN  LAST  TWO  YEARS  ACCEPTABLE 

WITH  CORRECTIONS 

E         LATEST  QSR  IN  LAST  TWO  YEARS  ACCEPTABLE 
WITH  CORRECTIONS 

F         LATEST  SPECIAL  SURVEY  IN  LAST  TWO  YEARS 
ACCEPTABLE  WITH  CORRECTIONS 

G  REJECT  RATE  6-14%  FOR  FIVE  OR  MORE 

LOTS/ANY  REJECT  RATE  FOR  LESS  THAN  FIVE 
LOTS  IN  LAST  TWO  YEARS 

H  ONE  UNSAT  FAT  IN  LAST  YEAR 

J  ONE  CATEGORY  "I"  QDR  IN  LAST  YEAR 

K  THREE-FIVE  CATEGORY  "II"  ACTION  QDRS  IN 

LAST  YEAR 

L  THREE-FIVE  CONTRACTOR  LIABLE  RODS  IN  LAST 

YEAR'' 

M  THREE-FIVE  CONTRACTOR  LIABLE  WAIVERS/ 

DEVIATIONS  IN  LAST  YEAR" 

N  ON  DLA  CONTRACTOR  ALERT  LIST  FOR  MINOR 

DEFICIENCIES'' 


V 


"classification  criteria  L,  M,  N  for  "Red"  and  "Yellow" 
have  not  yet  been  incorporated. 
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COLOR      CODE  CLASSIFICATION  CRITERIA 

GREEN       C       LATEST  PAS  IN  LAST  TWO  YEARS — AWARD  WITH 

NO  FINDINGS 

D       LATEST  POS  IN  LAST  TWO  YEARS  ACCEPTABLE 

E       LATEST  QDR  IN  LAST  TWO  YEARS  ACCEPTABLE 

F       LATEST  SPECIAL  SURVEY  IN  LAST  TWO  YEARS 
ACCEPTABLE 

G       REJECT  RATE  LESS  THAN  6%  IN  LAST  TWO  YEARS 
FOR  FIVE  OR  MORE  LOTS 

H       ALL  FAT  IN  LAST  YEAR  SATISFACTORY 

K       ONE-TWO  CATEGORY  "II"  ACTION  QDRS  IN  LAST 
YEAR 

L       ONE-TWO  CONTRACTOR  LIABLE  RODS  IN  LAST 
YEAR* 

M       ONE-TWO  CONTRACTOR  LIABLE  WAIVERS/ 
DEVIATIONS  IN  LAST  YEAR*' 


"'classification  criteria  L  and  M  for  "Green"  have  not  yet 
been  incorporated. 
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APPENDIX  B 
CLAUSES  FOR  SIMPLIFIED  SMALL  PURCHASE  PROCEDURES 

NOTICE  TO  PROSPECTIVE  OFFERORS  (NOV  1988) 

(a)  This  procurement  is  subject  to  a  test  of  the  Navy's 
Contractor  Evaluation  System  (CES) ,  "Red/Yellow/Green"  (R/Y/G) 
program.  The  test  is  authorized  by  the  Assistant  Secretary  of 
the  Navy  (Shipbuilding  and  Logistics)  for  the  acquisition  of 
specific  commodities  within  designated  Federal  Supply  Classes 
(FSCs)  by  participating  test  activities. 

(b)  The  Government  reserves  the  right  to  award  to  the 
contractor  whose  offer  represents  the  best  overall  purchase 
value  to  the  Government.  As  such,  the  basis  for  contract 
award  will  include  an  evaluation  of  proposed  contractors'  past 
quality  performance  history  on  the  particular  commodity  or 
commodities,  identified  below,  as  recorded  in  the  CES.  The 
price  to  be  considered  in  determining  best  value  will  be  the 
evaluated  price  after  Technical  Evaluation  Adjustments  (TEAs) 
for  related  quality  assurance  actions,  as  applicable,  are 
applied  to  the  offered  price. 

(c)  The  procedures  described  in  the  clause  of  this 
solicitation  entitled  "ADDITIONAL  EVALUATION  FACTORS — TEST  OF 
CONTRACTOR  EVALUATION  SYSTEM  (NOV  1988)"  will  be  used  by  the 
contracting  officer  to  assist  in  determining  the  best  purchase 
value  for  the  Government — price,  past  quality  performance,  and 
other  factors  considered. 

(d)  The  commodities  included  in  this  test,  as  currently 
solicited,  are: 

FSC  No.  FSC  Nomenclature  CLIN 
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ADDITIONAL  EVALUATION  FACTORS — TEST  OF  CONTRACTOR  EVALUATION 
SYSTEM  (NOV  1988)   (SIMPLIFIED  SMALL  PURCHASE  PROCEDURES) 

(a)  This  procurement  is  part  of  a  test  of  the  Navy's 
Contractor  Evaluation  System  (CES)  "Red/Yellow/Green"  (R/Y/G) 
Program,  authorized  by  the  Assistant  Secretary  of  the  Navy 
(Shipbuilding  and  Logistics) ,  for  the  acquisition  of  specific 
commodities  by  participating  activities.  At  the  end  of  the 
test,  data  concerning  awards  made  during  the  period  will  be 
evaluated  to  assess  the  program's  effectiveness  and  impact  on 
the  acquisition  process. 

(b)  The  purpose  of  R/Y/G  is  to  assist  contracting  personnel 
during  source  selection  to  determine  the  best  value  for  the 
Government--price,  past  quality  performance,  and  other  factors 
considered.  The  test  program  uses  accumulated  contractor 
quality  performance  data  to  classify  contractor's  performance 
on  select  commodities  as  either  "Red"  (high  risk) ,  "Yellow" 
(moderate  risk),  "Green"  (low  risk),  or  "Insufficient  Data," 
based  on  the  degree  of  risk  to  the  Government  of  receiving 
poor  quality  products.  Such  classifications  are  then  used  to 
apply  a  Technical  Evaluation  Adjustment  (TEA)  during  source 
selection. 

(c)  A  TEA  is  a  monetary  assessment  added  to  the  price  of 
selected  commodities  that  have  been  classified  as  either  "Red" 
or  "Yellow"  for  specific  contractors,  and  is  based  on  the  cost 
to  the  Government  for  effecting  additional  quality 
considerations  that  would  otherwise  not  be  required  if  award 
were  made  to  a  contractor  with  a  satisfactory  performance 
history.  For  purposes  of  requirements  using  the  simplified 
small  purchasing  procedures,  standardized  TEAs  have  been 
established  for  the  "Red"  and  the  "Yellow"  classifications. 
During  evaluation  of  quotations,  the  applicable  TEA  is  added 
to  the  quoted  price  of  the  "Red"  and/or  "Yellow"  commodity, 
and  after  consideration  of  any  other  pertinent  price-related 
factors  (e.g.,  transportation  charges.  First  Article  Testing, 
discount  terms,  etc.) ,  becomes  the  basis  for  determining  award 
of  the  purchase  order.  A  commodity's  classification  may 
change  over  time  as  new  or  revised  quality  performance  data 
become  available. 

(d)  Classifications  for  the  test  program  are  summarized  as 
follows : 

"Green"--Low  risk.  No  extraordinary  quality  requirements  or 
additional  actions  required;  satisfactory  quality  history. 

"Yellow" — Moderate  risk.  History  of  quality  problems;  special 
quality  requirements/actions  needed;  Technical  Evaluation 
Adjustment  (TEA)  applied  to  offered  price. 
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"Red" — High  risk.  Special  alert  to  history  of  poor  quality 
performance;  TEA  applied  to  offered  price (s),  and  contract 
award  requires  higher  level  approval. 

"Insufficient  Data" — Generally,  may  be  commodities  of  first- 
time  offerors  or  offerors  for  whom  current,  up-to-date  quality 
performance  history  is  unavailable;  additional  quality  actions 
may  be  needed  and  invoked;  however,  a  TEA  is  not  assessed. 

(e)  Prospective  offerors  may  address  questions  with  regard  to 
their  assessment  classification  on  particular  commodities  to: 
Naval  Sea  System  Command  Detachment,  Naval  Material  Quality 
Assessment  Office  (NMQAO) ,  Federal  Building,  Room  423,  80 
Daniel  Street,  Portsmouth,  NH  03801-3884,  (Telephone)  608- 
431-9460. 
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APPENDIX  C 
CLAUSES  FOR  MAJOR  PURCHASE  PROCEDURES 

NOTICE  TO  PROSPECTIVE  OFFERORS  (NOV  1988) 

(a)  This  procurement  is  subject  to  a  test  of  the  Navy's 
Contractor  Evaluation  System  (CES) ,  "Red/Yellow/Green"  (R/Y/G) 
program.  The  test  is  authorized  by  the  Assistant  Secretary  of 
the  Navy  (Shipbuilding  and  Logistics)  for  the  acquisition  of 
specific  commodities  within  designated  Federal  Supply  Classes 
(FSCs)  by  participating  test  activities. 

(b)  The  Government  reserves  the  right  to  award  to  the 
contractor  whose  offer  represents  the  best  overall  purchase 
value  to  the  Government.  As  such,  the  basis  for  contract 
award  will  include  an  evaluation  of  proposed  contractors'  past 
quality  performance  history  on  the  particular  commodity  or 
commodities,  identified  below,  as  recorded  in  the  CES.  The 
price  to  be  considered  in  determining  best  value  will  be  the 
evaluated  price  after  Technical  Evaluation  Adjustments  (TEAs) 
for  related  quality-assurance  actions,  as  applicable,  are 
applied  to  the  offered  price. 

(c)  The  procedures  described  in  the  clause  of  this 
solicitation  entitled  "ADDITIONAL  EVALUATION  FACTORS — TEST  OF 
CONTRACTOR  EVALUATION  SYSTEM  (NOV  1988)"  will  be  used  by  the 
contracting  officer  to  assist  in  determining  the  best  purchase 
value  for  the  Government — price,  past  quality  performance,  and 
other  factors  considered. 

(d)  The  commodities  included  in  this  test,  as  currently 
solicited,  are: 

FSC  No.  FSC  Nomenclature  CLIN 
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ADDITIONAL  EVALUATION  FACTORS — TEST  OF  CONTRACTOR  EVALUATION 
SYSTEM  (NOV  198  8)   (MAJOR  PURCHASE  PROCEDURES) 

(a)  This  procurement  is  part  of  a  test  of  the  Navy's 
Contractor  Evaluation  System  (CES)  "Red/Yellow/Green"  (R/Y/G) 
Program,  authorized  by  the  Assistant  Secretary  of  the  Navy 
(Shipbuilding  and  Logistics) ,  for  the  acquisition  of  specific 
commodities  by  participating  activities.  At  the  end  of  the 
test,  data  concerning  awards  made  during  the  period  will  be 
evaluated  to  assess  the  program's  effectiveness  and  impact  on 
the  acquisition  process. 

(b)  The  purpose  of  R/Y/G  is  to  assist  contracting  personnel 
during  source  selection  to  determine  the  best  value  for  the 
Government — price,  past  quality  performance,  and  other  factors 
considered.  The  test  program  uses  accumulated  contractor 
quality  performance  data  to  classify  contractor's  performance 
on  select  commodities  as  either  "Red"  (high  risk) ,  "Yellow" 
(moderate  risk),  "Green"  (low  risk),  or  "Insufficient  Data," 
based  on  the  degree  of  risk  to  the  Government  of  receiving 
poor  quality  products.  Such  classifications  are  then  used  to 
apply  a  Technical  Evaluation  Adjustment  (TEA)  during  source 
selection. 

(c)  A  TEA  is  a  monetary  assessment  added  to  the  price  of 
selected  commodities  that  have  been  classified  as  either  "Red" 
or  "Yellow"  for  specific  contractors,  and  is  based  on  the  cost 
to  the  Government  for  effecting  additional  quality 
considerations  that  would  otherwise  not  be  required  if  award 
were  made  to  a  contractor  with  a  satisfactory  performance 
history.  During  evaluation  of  quotations,  the  necessity  for 
any  additional  quality  assurance  requirements  will  be 
determined,  and  the  applicable  TEA  will  be  assessed  onto  the 
quoted  price  of  the  "Red"  and/or  "Yellow"  commodity.  After 
consideration  of  any  other  pertinent  price-related  factors 
(e.g.,  transportation  charges.  First  Article  Testing,  discount 
terms,  etc.),  this  adjusted  price  becomes  the  basis  for 
determining  award  of  the  purchase  order.  A  commodity's 
classification  may  change  over  time  as  new  or  revised  quality 
performance  data  become  available. 

(d)  Classifications  for  the  test  program  are  summarized  as 
follows : 

"Green" — Low  risk.  No  extraordinary  quality  requirements  or 
additional  actions  required;  satisfactory  quality  history. 

"Yellow" — Moderate  risk.  History  of  quality  problems;  special 
quality  requirements/actions  needed;  Technical  Evaluation 
Adjustment  (TEA)  applied  to  offered  price. 
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"Red" — High  risk.  Special  alert  to  history  of  poor  quality 
performance;  TEA  applied  to  offered  price(s),  and  contract 
award  requires  higher  level  approval. 

"Insufficient  Data" — Generally,  may  be  commodities  of  first- 
time  offerors  or  offerors  for  whom  current,  up-to-date  quality 
performance  history  is  unavailable;  additional  quality  actions 
may  be  needed  and  invoked;  however,  a  TEA  is  not  assessed. 

(e)  Prospective  offerors  may  address  questions  with  regard  to 
their  assessment  classification  on  particular  commodities  to: 
Naval  Sea  System  Command  Detachment,  Navy  Material  Quality 
Assessment  Office  (NMQAO) ,  Federal  Building,  Room  423,  80 
Daniel  Street,  Portsmouth,  NH  03801-3884,  (Telephone)  608- 
431-9460. 
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APPENDIX  D 

GUIDELINE  FOR  TEA  ASSIGNMENT 

RED  CLASSIFICATION 

CODE  CLASSIFICATION  CRITERIA 

A  On  Current  Navy  VDAR 

(Time  Frame — Six  Months) 


B  DLA  Corrective  Action 

Methods  C,  D,  or  E  Currently 

in  Effect 

(Time  Frame — Within  Two  Months) 


Latest  Pre-Award  Survey 
(Negative  Quality) 
(Time  Frame — Two  Years) 


Latest  Product  Oriented  Survey 
Rated  Unsat — Could  Impact 
Product  Quality 
(Time  Frame-Two  Years) 

Latest  Quality  System  Review 
Rated  Unsat — Could  Impact 
Product  Quality 
(Time  Frame — Two  Years) 


Latest  Quality  Audit/Special 

Survey 

Rated  Unsat — Could  Impact 

Product  Quality 

(Time  Frame — Two  Years) 


ADDITIONAL 

OA  REO'TS 

lA  or 

IB 

2A  or 

2B 

3 

4 

5  or 

6 

7 

lA  or 

IB 

2A  or 

2B 

4 

5  or 

6 

7 

lA  or 

IB 

4 

5  or 

6 

7 

lA  or 

IB 

4 

5  or 

6 

7 

lA  or 

IB 

2A  or 

2B 

4 

5  or 

6 

7 

lA  or 

IB 

4 

5  or 

6 

7 
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ADDITIONAL 


CODE  CLASSIFICATION  CRITERIA 

G  Material  Inspection  Records 

Reject  Rate  15%  or  Greater 
Indicated 
(Minimum  Five  Lots  in  Two  Years) 


H  First  Article  Test 

Two  or  More  Unsat 
(Time  Frame — One  Year) 


Quality  Deficiency  Reports — Cat  I 
Two  or  More  Unsat 
(Time  Frame — One  Year) 


K  Quality  Deficiency  Reports — 

Cat  II 

Six  or  More  "Action"  Cat  II 
(Time  Frame — One  Year) 


L  Report  of  Discrepancy 

Six  or  More  Contractor  Liable 

RODS 

(Time  Frame--One  Year) 

M  Waivers/Deviations 

Six  or  More  Contractor  Liable 

Requests 

(Time  Frame — One  Year) 

N  DLA  Contractor  Alert  List 

Listed  for  Major  Deficiencies 

Which  Could  Impact  Product 

Quality 

(Time  Frame--Six  Months) 


QA 

REO'TS 

lA 

or 

3 

4 

IB 

5 

or 

6 

5 

or 

7 

6 

lA 

or 

IB 

2A 

or 
4 

2B 

5 

or 

7 

6 

lA 

or 

IB 

2A 

or 
3 

4 

2B 

5 

or 

7 

6 

lA 

or 
3 

4 

IB 

5 

or 

7 

6 

2A 

or 

4 

2B 

5 

or 
7 

6 

2A 

or 

4 

2B 

5 

or 
7 

6 

lA 

or 

IB 

2A 

or 

3 

4 

2B 

5 

or 

7 

6 
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YELLOW  CLASSIFICATION 


CODE  CLASSIFICATION  CRITERIA 


OA  REO'TS 

lA  or 

IB 

3 

4 

5  or 

6 

7 

lA  or 

IB 

3 

4 

5  or 

6 

7 

lA  or 

IB 

4 

5  or 

6 

VDAR  Discussion  Contractor  or 
Letter  of  Concern  Issued 
(Time  Frame — Six  Months) 


B  Previously  Classified  "Red" 

(Time  Frame — Six  Months) 


C  Latest  Pre-Award  Survey 

Positive  for  Quality  but 
Negative  in 
One  or  More  Other  Categories 
(Time  Frame — Two  Years) 

D  Latest  Product  Oriented  Survey        4 

Unsat  for  Minor  Problems — No         5  or  6 
Impact  on  Product  Quality 
(Time  Frame — Two  Years) 

E  Latest  Quality  System  Review  4 

Unsat  For  Minor  Problems  5  or  6 

on  Product  Quality 
(Time  Frame — Two  Years) 

F  Latest  Quality  Audit/Special  4 

Survey  5  or  6 

Unsat  for  Minor  Problems — No 
Impact  on  Product  Quality 
(Time  Frame — Two  Years) 

G  Material  Inspection  Reports  4 

Reject  Rate  Greater  Than  5%  5  or  6 

but  less  than  15%  Indicated  7 

(Based  on  Minimum  of  Five  Lots 
in  Two  Years) 

H  First  Article  Test  2A 

One  Unsat  4 

(Time  Frame — One  Year)  5  or  6 

7 
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CODE  CLASSIFICATION  CRITERIA 

J  Quality  Deficiency  Reports — Cat  I 

One  Cat  I 
(Time  Frame — One  Year) 

K  Quality  Deficiency  Reports--Cat  II 

More  Than  Two  but  Less  Than  Six 
Cat  II 
(Time  Frame — One  Year) 

L  Report  of  Discrepancy 

More  than  Two  but  Less  Than  Six 
Contractor  Liable  RODs 
(Time  Frame — One  Year) 

M  Waivers/Deviations 

More  Than  Two  but  Less  Than  Six 
Contractor  Liable  Requests 
(Time  Frame — One  Year) 

N  DLA  Contractor  Alert  List 

Listed  for  Minor  Deficiencies — 
No  Impact  on  Product  Quality 
(Time  Frame — One  Year) 


ADDITIONAL 
OA  REO'TS 

4 
5  or 

7 

6 

4 
5  or 

7 

6 

2A 

4 

7 

2A 
4 

7 

2A 

4 
5  or 

6 
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ADDITIONAL  QUALITY  ASSURANCE  REQUIREMENTS 

Code  and  Description  TEA 

1.  Pre-Award  Survey 

A.  DCAS — Quality  Survey  $   500 

B.  With  procurement  representative 
participation 

(1)  Local^  775 

(2)  Intermediate^  1,380 

(3)  Distant^  2,095 

2.  Post-Award  Orientation  Conference 

A.  DCAS — Quality  550 

B.  With  procurement  representative 
participation 

(1)  Local"*  1,075 

(2)  Intermediate^  2,110 

(3)  Distant^  3,590 

3  .     Product  Oriented  Survey — Procurement 

Representative  with  DCAS  participation 

(1)  Local^  800 

(2)  Intermediate®  1,500 

(3)  Distant^  2,215 

4.  Government  Source  Inspection'^"  500 

5.  Receipt  Inspection  at  Source--Navy 
representative  with  DCAS  participation 

(1)  Local^^  650 

(2)  Intermediate^^  1,3  60 

(3)  Distant^^  2,182 

6.  Receipt  Inspection  at  Destination--Navy 
representative 

(1)  Low^'  597 

(2)  Medium^^  1,194 

(3)  High^^  2,332 

7.  Quality  Assurance  Letter  of  Instruction'^''  755 

8.  Purchase  Referrals^®  640 
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TEA  CALCULATIONS 

^Calculated  $30/hr  x  8  hrs  =  $240  +  $35  mileage  =  $275  + 
500. 

^Calculated  $30/hr  x  8  hrs  =  $240  +  $240  (8  hrs  travel  @ 
$30/hr)  +  $200  (2  days  per  diem  @  $100/day)  +  $200  travel 
costs  =  $880  +  $500. 

^Calculated  $30/hr  x  8  hrs  =  $240  +  $480  (16  hrs  travel  @ 
$30/hr)  +  $300  (  3  days  per  diem  @  $100/day)  =  $575  travel 
costs  =  $1,595  +  $500. 

''Calculated  $30/hr  x  16  hrs  =  $480  +  $45  mileage  =  $525  + 
$550. 

^Calculated  $30/hr  x  16  hrs  =  $480  +  $480  (16  hrs  travel  @ 
$30/hr)  +  $400  (4  days  per  diem  @  $100/day)  +  $300  travel 
costs  =  $1,660  +  $550. 

^Calculated  $30/hr  x  16  hrs  =  $480  +  $960  (32  hrs  travel  @ 
$30/hr)  +  $600  (6  days  per  diem  @  $100/day)  +  $1,000  travel 
costs  =  $3,040  +  $550. 

^Calculated  $30/hr  x  12  hrs  =  $360  +  $40  mileage  =  $400  + 
$400  (DCAS  costs) . 

^Calculated  $30/hr  x  12  hrs  =  $360  +  $240  (8  hrs  travel  (a 
$30/hr)  +  $300  (  3  days  per  diem  (a  $100/day)  +  $200  travel 
costs  =  $1,100  +  $400  (DCAS  costs). 

^Calculated  $30/hr  x  12  hrs  =  $360  +  $480  (16  hrs  travel  @ 
$30/hr)  +  $400  (4  days  per  diem  §  $100/day)  +  $575  travel 
costs  =  $1,815  +  $400  (DCAS  costs). 

^"calculated  $34.18/hr  x  14  hrs. 

^^Calculated  $43/hr  x  8  hrs  =  $344  +  $31  mileage  =  $375  + 
$275  (DCAS  costs) . 

^^Calculated  $43/hr  x  8  hrs  =  $344  +  $344  (8  hrs  travel  @ 
$43/hr)  +  $200  (  2  days  per  diem  @  $100/day)  +  $200  travel 
costs  =  $1,088  +  $275  (DCAS  costs). 

^^Calculated  $43/hr  x  8  hrs  =  $344  +  $688  (16  hrs  travel  @ 
$43/hr)  +  $300  (  3  days  per  diem  @  $100/day)  +  $575  travel 
costs  =  $1,907  +  @275  (DCAS  costs). 

^'calculated  $43/hr  x  4  hrs  =  $172  +  $100  material  handling  + 
$325  test. 

^^Calculated  $43/hr  x  8  hrs  =  $344  +  $200  material  handling  + 
$650  test. 
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^^Calculated  $43/hr  x  24  hrs  =  $1,032  +  $500  material 
handling  +  $800  test. 

^'calculated  DCAS  @  $34.18/hr  x  8  hrs  =  $275  +  $480 
(procurement  representative  @  $30/hr  x  16  hrs) . 

^^Calculated  procurement  representative  @  $40/hr  x  16  hrs 
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APPENDIX  E 
CLAUSES  FOR  FIXED  PRICE — GREATEST  VALUE  PROCEDURES 

NOTICE  TO  PROSPECTIVE  OFFERORS  (NOV  1988) 

(a)  This  procurement  is  subject  to  a  test  of  the  Navy's 
Contractor  Evaluation  System  (CES)  "Red/Yellow/Green"  (R/Y/G) 
program.  The  test  is  authorized  by  the  Assistant  Secretary  of 
the  Navy  (Shipbuilding  and  Logistics)  for  the  acquisition  of 
specific  commodities  within  designated  Federal  Supply  Classes 
(FSCs)  by  participating  test  activities. 

(b)  The  Government  reserves  the  right  to  award  to  the 
contractor  whose  offer  represents  the  best  overall  purchase 
value  to  the  Government.  As  such,  the  basis  for  contract 
award  will  include  an  evaluation  of  proposed  contractors'  past 
quality  performance  history  on  the  particular  commodity  or 
commodities,  identified  below,  as  recorded  in  the  CES. 

(c)  The  procedures  described  in  the  clause  of  this 
solicitation  entitled  "ADDITIONAL  EVALUATION  FACTORS — TEST  OF 
CONTRACTOR  EVALUATION  SYSTEM  (NOV  1988)"  will  be  used  by  the 
contracting  officer  to  assist  in  determining  the  best  purchase 
value  for  the  Government — price,  past  quality  performance,  and 
other  factors  considered. 

(d)  The  commodities  included  in  this  test,  as  currently 
solicited,  are: 

FSCNp.       FSC  Nomenclature  CLIN 
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ADDITIONAL  EVALUATION  FACTORS — TEST  OF  CONTRACTOR  EVALUATION 
SYSTEM  (NOV  1988)   (FIXED  PRICE-GREATEST  VALUE  PROCEDURES) 

(a)  This  procurement  is  part  of  a  test  of  the  Navy's 
Contractor  Evaluation  System  (CES)  "Red/Yellow/Green"  (R/Y/G) 
Program,  authorized  by  the  Assistant  Secretary  of  the  Navy 
(Shipbuilding  and  Logistics) ,  for  the  acquisition  of  specific 
commodities  by  participating  activities.  At  the  end  of  the 
test,  data  concerning  awards  made  during  the  period  will  be 
evaluated  to  assess  the  program's  effectiveness  and  impact  on 
the  acquisition  process. 

(b)  The  purpose  of  R/Y/G  is  to  assist  contracting  personnel 
during  source  selection  to  determine  the  best  value  for  the 
Government — price,  past  quality  performance,  and  other  factors 
considered.  The  test  program  uses  accumulated  contractor 
quality  performance  data  to  classify  contractor's  performance 
on  select  commodities  as  either  "Red"  (high  risk) ,  "Yellow" 
(moderate  risk),  "Green"  (low  risk),  or  "Insufficient  Data," 
based  on  the  degree  of  risk  to  the  Government  of  receiving 
poor  quality  products.  A  commodity's  classification  may 
change  over  time  as  new  or  revised  quality  performance  data 
become  available. 

(c)  For  purposes  of  source  evaluation  and  selection,  both  the 
color  classification  of  an  offeror's  commodity  and  the 
proposed  price (s)  shall  be  evaluated  in  accordance  with 
weighted  evaluation  criteria  established  by  the  Government 
prior  to  the  receipt  of  proposals.  Price-related  factors, 
such  as  transportation  charges,  discount  terms,  the  cost  of 
First  Article  Test,  etc.,  will  also  be  considered;  however, 
no  score  or  rating  shall  be  applied. 

(d)  Offerors  are  advised  that,  although  price  is  of 
significance  in  determining  the  successful  offeror,  past 
quality  performance  on  the  proposed  commodity  (as  classified 
with  the  R-Y-G  database)  is  essentially  more  important,  and 
shall  be  evaluated  accordingly.  Each  of  the  R-Y-G 
classifications  and  its  relative  order  of  importance  is 
summarized  as  follows: 

"Green" — Low  risk.  No  extraordinary  quality  requirements  or 
additional  actions  required;  satisfactory  quality  history. 
Commodities  within  this  classification  are  apportioned  a 
greater  weight  or  value  in  the  evaluation  than  those 
classified  as  either  "Yellow"  or  "Red." 

"Yellow" — Moderate  risk.  History  of  quality  problems;  special 
quality  requirements/actions  may  be  needed.  Due  to  the 
additional  quality  assurance  considerations  that  may  be 
necessary,  commodities  within  this  classification  are  weighted 
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less  than  those  classified  as  "Green",  but  are  of  greater 
value  than  those  within  the  "Red"  category. 

"Red" — High  risk.  Special  alert  to  history  of  poor  quality 
performance;  contract  award  requires  higher  level  approval. 
These  commodities  are  apportioned  the  least  available  weight 
or  value  for  past  quality  performance  relative  to  commodities 
within  the  "Green"  or  "Yellow"  classifications. 

"Insufficient  Data" — Generally,  may  be  commodities  of  first- 
time  offerors  or  offerors  for  whom  current,  up-to-date  quality 
performance  history  is  unavailable;  additional  quality  actions 
may  be  needed  and  invoked;  however,  commodities  within  this 
classification  shall  be  evaluated  solely  on  the  basis  of  price 
and  related  factors.  Past  quality  performance  will  not  be  a 
consideration  in  the  evaluation  of  commodities  for  which 
current  quality  performance  data  is  not  set  forth  within  the 
database. 

(e)  Prospective  offerors  may  address  questions  with  regard  to 
their  assessment  classification  on  particular  commodities  to; 
Naval  Sea  System  Command  Detachment,  Navy  Material  Quality 
Assessment  Office  (NMQAO) ,  Federal  Building,  Room  423,  80 
Daniel  Street,  Portsmouth,  NH  03801-3884,  (Telephone)  608- 
431-9460. 
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APPENDIX  F 

SAMPLE  EVALUATION  UNDER  FIXED  PRICE — GREATEST 
VALUE  PROCEDURES 


Source  Selection/Evaluation  Method 

Total  Points  (Maximum)       =  100  points  (%) 

Total  Technical  =60  points  (%) 

Price  =40  points  (%) 

Classification         Adjective       Numerical  Rating 

Green  Excellent  55-60  points 


II 


Good  40-54  points 


Yellow  Good  35-39  points 

"  Average  25-34  points 

"  Marginal  16-24  points 

Red  Marginal  10-15  points 

"  Unacceptable  0-14  points 


Evaluation  Measurements 
Yellow  Classification  Adjective       Numerical  Rating 

Reject  Rate  =  6%-8%     Good  35-39  points 

(Code  G) 
Reject  Rate  =  9%-ll%    Average  25-34  points 

(Code  G) 
Reject  Rate  =  12%-14%   Marginal  16-24  points 

(Code  G) 

3  Category  "II" 

action  QDRs  Good  35-39  points 

(Code  K) 

4  Category  "II" 

action  QDRs  Average  25-34  points 

(Code  K) 

5  Category  "II" 

action  QDRs  Marginal  16-24  points 

(Code  K) 
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